Investigating 1.7.1 vs 1.8.0 vs 1.8.1

[slinlee]

1.7.1 was pre-reward terms
1.8.0 introduced reward terms and reward balancing while maintaining compatibility with classic reward functions. Found to have worse performing policies in some cases
1.8.1 - enhances reward balancing and fixes some bugs.

1.8.1 is available on test.devpathmind.com, but right now we need to update the webapp to use the new 'use_auto_norm' flag

results are on this spreadsheet https://docs.google.com/spreadsheets/d/1LUpvfqk9yLMuoGngS9JZ86Oro3jSoC7bhqW3_XLRqTg/edit?usp=sharing

--- old info ---
1.7.1 - agv - classic reward function https://test.devpathmind.com/sharedExperiment/7049 the results look terrible. Throughput is 2. I think the webapp's format for setting up trainings isn't compatible with 1.7.1 anymore. (attn @ejunprung so we'll test locally??)
1.8.1 - agv - classic reward function https://test.devpathmind.com/experiment/7044

[slinlee]

I'll be updating this as I continue testing. @ejunprung

[ejunprung]

👍 I should be able to start testing as well soon.

[SaharEs]

I am testing Field Service, Amazon's model, and Felipe's model.

[ejunprung]

Btw, here is the Product Delivery results.

NativeRL 1.8.0 - https://app.pathmind.com/sharedExperiment/17265
NativeRL 1.7.1 - https://app.pathmind.com/sharedExperiment/17266

1.8.0 works well but in 1.7.1, the reward is cleared out completely (i.e. all 0s). @slinlee Do you happen to know where the reward terms stuff is in the webapp code?

[slinlee]

Yeah. Using the current webapp will send an empty reward function where 1.7.1 expects it. So that doesn't work unfortunately.

[ejunprung]

Okay, I'm gonna do a local training now to confirm if (2) is actually a problem or not.

[ejunprung]

@slinlee Weird, in my local, the reward is all 0s as well. Are you sure the webapp clears it out? Or is it something else?

[slinlee]

you were testing 1.7.1? oh I saw your message in slack just now.

[ejunprung]

Yup

[slinlee]

Update: I created a spreadsheet because editing the table above sucked. https://docs.google.com/spreadsheets/d/1LUpvfqk9yLMuoGngS9JZ86Oro3jSoC7bhqW3_XLRqTg/edit?usp=sharing