Use 4-char element ids instead of sequential integers #361

alexzarbn · 2024-05-24T10:37:53Z

Problem

This issue with element ids is tightly related to the iframes support. When element tree is built by injecting domUtils.js into each iframe, the resulting ids are being messed up, because at first the page element tree is built and then element tree of each iframe on that page, resulting in iframe children ids duplicating some ids of elements on the page.

Solution

Use random 4-char ids for elements. This way the we can manipulate the tree without worrying about uniqueness of the element ids.

skyvern/webeye/scraper/domUtils.js

ykeremy

Looks great! Tested on my local as well, worked really well. I've couple of small comments.

skyvern/webeye/scraper/domUtils.js

ykeremy · 2024-05-26T00:12:25Z

There are some prettier and mypy errors as well, would be perfect if you could take a look at those.

…access in O(1) instead

alexzarbn · 2024-05-28T07:43:39Z

@ykeremy

There are some prettier and mypy errors as well, would be perfect if you could take a look at those.

Fixed those, but for this mypy error I had to add unique ids to options too. Not sure, if this is the right course of action in this case.

skyvern/webeye/actions/handler.py:487: error: Invalid index type "int" for "dict[str, str]"; expected type "str"  [index]

Please met know, what was the intention behind mapping index -> id in the prompt, whether it was a typo (also considering that there is no optionIndex included in the prompt as far as I can see) or it was intentional, and whether including options ids in the prompt is okay. Perhaps the implementation should rely solely on option indexes and match by xpath could be removed?

"index": int, // the id corresponding to the optionIndex under the the select element.

LawyZheng · 2024-05-31T11:29:43Z

skyvern/forge/prompts/skyvern/extract-action.j2

            "value": str // the value of the option. MAKE SURE YOU USE THIS VALUE TO SELECT THE OPTION. DO NOT PUT ANYTHING OTHER THAN A VALID OPTION VALUE HERE
+            "id": str // the id of the option.


Can we remove the option.id to prevent LLM hallucinating.

Yes, we can remove this. I don't think we're even passing the option id to the LLM.

LawyZheng · 2024-05-31T11:34:56Z

skyvern/webeye/actions/handler.py

@@ -484,7 +484,7 @@ async def handle_select_option_action(
            return [ActionFailure(e)]

    try:
-        option_xpath = scraped_page.id_to_xpath_dict[action.option.index]
+        option_xpath = scraped_page.id_to_xpath_dict[action.option.id]


Actually this is a legacy bug. We didn't store the option element as a single key in the scraped_page.id_to_xpath_dict. So the option.id here will never be found.

skyvern/skyvern/webeye/scraper/scraper.py

Lines 257 to 263 in 418473d

id_to_xpath_dict = {}

id_to_element_dict = {}

for element in elements:

element_id = element["id"]

# get_interactable_element_tree marks each interactable element with a unique_id attribute

id_to_xpath_dict[element_id] = f"//*[@{SKYVERN_ID_ATTR}='{element_id}']"

id_to_element_dict[element_id] = element

Oh yeah this is a big bug 😮 We should stop using id_to_xpath dict and deprecate it IMO

LawyZheng · 2024-05-31T11:40:49Z

this can trigger an error. Can you remove this?

skyvern/skyvern/webeye/actions/handler.py

Line 776 in 418473d

element_id = int(element_id)

LawyZheng · 2024-05-31T11:46:14Z

@alexzarbn

Please met know, what was the intention behind mapping index -> id in the prompt, whether it was a typo (also considering that there is no optionIndex included in the prompt as far as I can see) or it was intentional, and whether including options ids in the prompt is okay. Perhaps the implementation should rely solely on option indexes and match by xpath could be removed?
"index": int, // the id corresponding to the optionIndex under the the select element.

nice catch. this is a legacy problem after we finished the HTML element tree.
can you also fix this in the extract-action-claude3-sonnet.j2 file?

LawyZheng · 2024-05-31T11:49:37Z

after you finish all suggestions, can you solve the conflict?
and we can start the testing.

ykeremy · 2024-06-03T06:57:36Z

@LawyZheng we took over this PR, feel free to merge this on a new branch and then take over.

ykeremy · 2024-06-03T07:09:36Z

skyvern/forge/prompts/skyvern/extract-action.j2

        "text": str, // Text for INPUT_TEXT action only
        "file_url": str, // The url of the file to upload if applicable. This field must be present for UPLOAD_FILE but can also be present for CLICK only if the click is to upload the file. It should be null otherwise.
        "download": bool, // Can only be true for CLICK actions. If true, the browser will trigger a download by clicking the element. If false, the browser will click the element without triggering a download.
        "option": {  // The option to select for SELECT_OPTION action only. null if not SELECT_OPTION action
            "label": str, // the label of the option if any. MAKE SURE YOU USE THIS LABEL TO SELECT THE OPTION. DO NOT PUT ANYTHING OTHER THAN A VALID OPTION LABEL HERE
-            "index": int, // the id corresponding to the optionIndex under the the select element.
+            "index": int, // the index corresponding to the index under the select element.


Suggested change

"index": int, // the index corresponding to the index under the select element.

"index": int, // the index corresponding to the option index under the select element.

We add a specific option index field to the element tree. Let's keep this wording the same.

ykeremy · 2024-06-03T07:13:00Z

skyvern/forge/prompts/skyvern/extract-action.j2

            "value": str // the value of the option. MAKE SURE YOU USE THIS VALUE TO SELECT THE OPTION. DO NOT PUT ANYTHING OTHER THAN A VALID OPTION VALUE HERE
+            "id": str // the id of the option.


Yes, we can remove this. I don't think we're even passing the option id to the LLM.

ykeremy · 2024-06-03T07:13:30Z

skyvern/webeye/actions/actions.py

@@ -93,9 +93,10 @@ class SelectOption(BaseModel):
    label: str | None
    value: str | None
    index: int | None
+    id: str | None


@LawyZheng let's remove this too

ykeremy · 2024-06-03T07:17:37Z

skyvern/webeye/actions/handler.py

@@ -484,7 +484,7 @@ async def handle_select_option_action(
            return [ActionFailure(e)]

    try:
-        option_xpath = scraped_page.id_to_xpath_dict[action.option.index]
+        option_xpath = scraped_page.id_to_xpath_dict[action.option.id]


Oh yeah this is a big bug 😮 We should stop using id_to_xpath dict and deprecate it IMO

Co-authored-by: LawyZheng <lawyzheng1106@gmail.com>

feat: 4-char element ids

e48e8c0

alexzarbn mentioned this pull request May 24, 2024

iFrames support #362

Closed

suchintan reviewed May 24, 2024

View reviewed changes

skyvern/webeye/scraper/domUtils.js Outdated Show resolved Hide resolved

ykeremy approved these changes May 26, 2024

View reviewed changes

skyvern/webeye/scraper/domUtils.js Outdated Show resolved Hide resolved

skyvern/webeye/scraper/domUtils.js Show resolved Hide resolved

skyvern/webeye/scraper/domUtils.js Outdated Show resolved Hide resolved

alexzarbn added 2 commits May 28, 2024 12:31

fix: handling select options via xpath, code styling issues

83b4258

chore: add todo about using dict/object so that element children are …

9daf02d

…access in O(1) instead

alexzarbn requested a review from ykeremy May 28, 2024 07:44

LawyZheng reviewed May 31, 2024

View reviewed changes

ykeremy reviewed Jun 3, 2024

View reviewed changes

LawyZheng added 5 commits June 3, 2024 15:37

Merge branch 'main' into feature/4-char-element-ids

b921f6d

solve pre-commit failure

f90a2cb

remove select.option.id

3324db1

fix sonnet prompt bug

efb42e0

remove select.option.id

a788181

LawyZheng merged commit d5abfc9 into Skyvern-AI:main Jun 3, 2024
2 checks passed

jesalg pushed a commit to Thoughtful-Automation/skyvern that referenced this pull request Jun 13, 2024

Use 4-char element ids instead of sequential integers (Skyvern-AI#361)

c90f49e

Co-authored-by: LawyZheng <lawyzheng1106@gmail.com>

jesalg pushed a commit to Thoughtful-Automation/skyvern that referenced this pull request Jun 13, 2024

Use 4-char element ids instead of sequential integers (Skyvern-AI#361)

1d43897

Co-authored-by: LawyZheng <lawyzheng1106@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use 4-char element ids instead of sequential integers #361

Use 4-char element ids instead of sequential integers #361

alexzarbn commented May 24, 2024

ykeremy left a comment

ykeremy commented May 26, 2024

alexzarbn commented May 28, 2024 •

edited

Loading

LawyZheng May 31, 2024

ykeremy Jun 3, 2024

LawyZheng May 31, 2024

ykeremy Jun 3, 2024

LawyZheng commented May 31, 2024

LawyZheng commented May 31, 2024 •

edited

Loading

LawyZheng commented May 31, 2024

ykeremy commented Jun 3, 2024

ykeremy Jun 3, 2024

ykeremy Jun 3, 2024

ykeremy Jun 3, 2024

ykeremy Jun 3, 2024

		"value": str // the value of the option. MAKE SURE YOU USE THIS VALUE TO SELECT THE OPTION. DO NOT PUT ANYTHING OTHER THAN A VALID OPTION VALUE HERE
		"id": str // the id of the option.

	id_to_xpath_dict = {}
	id_to_element_dict = {}
	for element in elements:
	element_id = element["id"]
	# get_interactable_element_tree marks each interactable element with a unique_id attribute
	id_to_xpath_dict[element_id] = f"//*[@{SKYVERN_ID_ATTR}='{element_id}']"
	id_to_element_dict[element_id] = element

	"index": int, // the index corresponding to the index under the select element.
	"index": int, // the index corresponding to the option index under the select element.

Use 4-char element ids instead of sequential integers #361

Use 4-char element ids instead of sequential integers #361

Conversation

alexzarbn commented May 24, 2024

Problem

Solution

ykeremy left a comment

Choose a reason for hiding this comment

ykeremy commented May 26, 2024

alexzarbn commented May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LawyZheng commented May 31, 2024

LawyZheng commented May 31, 2024 • edited Loading

LawyZheng commented May 31, 2024

ykeremy commented Jun 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexzarbn commented May 28, 2024 •

edited

Loading

LawyZheng commented May 31, 2024 •

edited

Loading