I got zero score running Webshop.ipython

[yhao-wang]

I tried to run Webshop.ipython, and here are some of the outputs:

Observation: Invalid action!

Action: click[Add to Cart]
Observation: 

Action: click[Add to Cart]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

1 0.0 0.0 0.0
-------------
-----------------
1
Action: reset
Observation: 

Action: click[Buy Now]
Observation: Invalid action!

Action: click[Add to Cart]
Observation: 

Action: click[Add to Cart]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: click[Proceed to checkout]
Observation: 

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

Action: fill out form[name: John Doe, email: john@doe.com, phone: 555-555-1212, address: 123 Main St, city: Anytown, state: CA, zip: 99999]
Observation: Invalid action!

2 0.0 0.0 0.0
-------------
-----------------

How can I get the right score? Thank you

[yhao-wang]

By the way, could you tell me the score of GPT-3 on webshop? Thanks a lot.

[ysymyth]

hi, can you tell me your exact commands? did you exactly run the Jupyter notebook?

[Hykaruu]

hi, can you tell me your exact commands? did you exactly run the Jupyter notebook?

Hello, it looks like the live website from https://webshop-pnlp.github.io/ (http://3.83.245.205:3000/) used in the notebook is down. Could you help me check it?

[yhao-wang]

I guess it is down. I received 'HTTP Error 502'. Is there any way for me to run the codes without accessing live website? Thank you.

[ysymyth]

thanks @yhao-wang for letting us know!

I think the better choice is to use the webshop package at https://github.com/princeton-nlp/webshop instead of scraping the website. I might need some time to implement it, but would be more than happy to pull your request if you try it out.

[yhao-wang]

I tried to use webshop/web_agent_site/envs/web_agent_text_env.py as env, and i had a few problems. So far, I found the env does not give feedback like 'invalid action!'. How to change the codes to get the same outputs as ReAct? Are there any other difference? Thanks!

[ysymyth]

what did you get exactly when you issue an invalid action?

[ysymyth]

@yhao-wang since it's been two weeks, close for now. feel free to open a new one with more concrete logs