Feature Request: improvement for local evaluation

[samuelcai-chancetop]

Can the SDK be set to not get the flag configuration from the server at start up?

We have a use case to get variant right after app startup, so we would like to use the Local Evaluation because of the speed, but it still depends on flag configurations getting from the server, more over the flag getting makes the app start up depends on Amplitude server.
So here is what I think if the SDK can do this:

1. Similar to fetchOnStart, there is a getFlagOnStart which by default is true.
2. If getFlagOnStart is false, user needs to getFlag after start.
3. User can set some initialFlags for local evaluation, I see there's one pull request already: feat: bootstrap initial local evaluation flags #36

One thing I am not sure is, does this require Amplitude Experiment backend and website support?

[bgiori]

Hi @samuelcai-chancetop thanks for submitting this issue, and sorry for the delayed response. We're working on updating that stale PR so that the SDK can indicate that it has loaded flags from the async storage without needing to call start/fetch

For example, you will be able to await a new ready promise on the client which resolves when the SDK has completed loading the flags from storage:

const experiment = await Experiment.initialize('API_KEY', { initialFlags: 'INITIAL_FLAGS' }).ready;
const variant = experiment.variant('my-local-eval-flag'); // this evaluates successfully using flags loaded in config

Would this work for you? Let me know if I am misunderstanding your request.

[samuelcai-chancetop]

Hi @bgiori Yes that works for me, if I understand correctly:

const experiment = await Experiment.initialize('API_KEY', { initialFlags: 'INITIAL_FLAGS' }).ready;
const variant = experiment.variant('my-local-eval-flag'); // this evaluates successfully using flags loaded in config
experiment.getFlags(); // new method to load flags async, so no blocking app start. The loaded flags will overwrite the initial flags.
experiment.fetch(); // fetch variants async, so no blocking app start

Currently the local evaluation flags are setup in Amplitude Experiment server first and loaded in experiment.start(), with the new change, for { initialFlags: 'INITIAL_FLAGS' }, do I still need to setup the INITIAL_FLAGS in Amplitude Experiment server first (same key but can have different segments)?

[tyiuhc]

Hi @samuelcai-chancetop, in v1.5.0, you can now make use of the initialFlags config option. For your use case:

1. Use curl command to get flags configurations, and set initialFlags upon initialization of the experiment client.
   curl https://flag.lab.amplitude.com/sdk/v2/flags?v=0 -H 'Authorization: Api-Key DEPLOYMENT_KEY'
2. Await cacheReady() after init to ensure variants and flags have been loaded from async storage
3. Call start() without await to do async fetch of variants and flags from server

As an example:

const experiment = Experiment.initializeWithAmplitudeAnalytics('deployment', {
	initialFlags: 'INITIAL_FLAGS'
});
await experiment.cacheReady();
experiment.start();

On your question: No, you do not need to setup "INITIAL_FLAGS" in Amplitude Experiment server first before using a flag via the initialFlags config.

Please let us know if anything is unclear.

[samuelcai-chancetop]

Hi @tyiuhc , thanks for the update, that's really useful, I have three more questions:

1. About "you do not need to setup 'INITIAL_FLAGS' in Amplitude Experiment server first". If some flags in initialFlags do not exist on Experiment server, will the SDK help create them onto the server, with your example? I did some test but did not see that happen. I thought the procedure is creating the flags first on server, and then use them in codes?
2. I tried using the curl command and it downloaded all flags, including remote evaluation, can I just cherry pick the local ones? Did a test, seems ok.
3. Can I call experiment.variant("local flag") right after await experiment.cacheReady(), before experiment.start()? I did some test, always return undefined. This does not meet my requirement, because I thought that I had already init the local flags, I wanted to use it right away, did not want to wait for start() method end, because start() method requires network call. Am I misunderstanding something?

[tyiuhc]

@samuelcai-chancetop, regarding your questions.

1. Apologies if my answer was unclear. If the flags in initialFlags do not exist on the server, they WILL NOT be created. However, you will still be able to assign variants based on these flags. Yes, the standard practice should be to create flags in Experiment first.
2. Yes
3. On this, you should not have to call start() at all, as long as initialFlags for local flag is set in the config, and the apiKey is not an empty string.

Hope this helps!

[samuelcai-chancetop]

@tyiuhc , thanks, most are clear, for my question 3, I think I found the root cause, here is an example

Analytics.init(amplitudeApiKey);
const experiment = Experiment.initializeWithAmplitudeAnalytics('deployment', {
	initialFlags: 'LOCAL_FLAGS' // local flags which use device id for bucketing
});
await experiment.cacheReady();
console.log("Amplitude Device Id: " + Analytics.getDeviceId()); // undefined here, need sometime to init or load from cache
experiment.variant('LOCAL_FLAG');

When getting variant for local evaluation, if the device id is undefined yet, then the variant is undefined.
If I wait until Analytics.getDeviceId() has value and then get the variant, then no problem.

This is not an issue for fetch: the local evaluation method evaluate relies on method addContextSync, but the doFetch method relies on addContextOrWait which waits for identity (including device id) ready.