Add hinting support for Shadow DOM nested elements #3001

heiwiper · 2023-06-04T21:13:46Z

Description

Shadow DOM nested elements are not currently handled by the hinting feature. This pull request aims to add support for these type of elements by expanding the search for hintable elements through Shadow DOM trees.

Fixes #1828

Discussion

I tried as much as I can to introduce less code additions and changes but I had to use the new macros rqs and rqsa instead of qs and qsa respectively in mode/hint.lisp and dom.lisp (Please keep in mind that for the replacements I made in dom.lisp I mainly tested the function click-element and then added the same changes to the functions that are similar like check-element, I wasn't sure how to test the rest and I did not want to waste more time figuring that out so I thought I would add this working code and confirm the other functions are working as expected).

I would also like to precise that this pull request was mostly inspired by #2821 in the beginning due to the similarity between Shadow DOM and IFrame elements, but then I had to change most of the code due to some changes that introduced bugs like the one I have mentioned here. One of the changes that I had to drop was adding multiple stylesheets such as in the commit from the pull request I linked earlier.

I also want to suggest moving all the hint elements into a single parent element outside the main DOM body, the current behavior is appending the hint overlay elements to the body which makes it very cluttered and we would have to remove them one by one which wouldn't be the case if we had all elements under the same custom nyxt node, this is the case for saka-key and LinkHints .

Checklist:

Git hygiene:
- I have pulled from master before submitting this PR
- There are no merge conflicts.

I've added the new dependencies as:

ASDF dependencies,

Git submodules,

cd /path/to/nyxt/checkout
git submodule add https://gitlab.common-lisp.net/nyxt/py-configparser _build/py-configparser

and Guix dependencies.

My code follows the style guidelines for Common Lisp code. See:
- Norvig & Pitman's Tutorial on Good Lisp Programming Style (PDF)
- Google Common Lisp Style Guide
I have performed a self-review of my own code.
My code has been reviewed by at least one peer. (The peer review to approve a PR counts. The reviewer must download and test the code.)
Documentation:
- All my code has docstrings and :documentations written in the aforementioned style. (It's OK to skip the docstring for really trivial parts.)
- I have updated the existing documentation to match my changes.
- I have commented my code in hard-to-understand areas.
- I have updated the changelog.lisp with my changes if it's anything user-facing (new features, important bug fix, compatibility breakage).
- I have added a migration.lisp entry for all compatibility-breaking changes.
- (If this changes something about the features showcased on Nyxt website) I have these changes described in the new/existing article at Nyxt website or will notify one of maintainters to do so.
Compilation and tests:
- My changes generate no new warnings.
- I have added tests that prove my fix is effective or that my feature works. (If possible.)
- I ran the tests locally ((asdf:test-system :nyxt) and (asdf:test-system :nyxt/gi-gtk)) and they pass.

heiwiper · 2023-06-04T23:06:42Z

I have just tested the following interactive functions in dom.lisp, the checked elements are work as expected:

click-element
check-element
focus-select-element
toggle-details-element
select-option-element
scroll-to-element ~~(I didn't know how to test this function)~~
set-caret-on-start (Generates errors because visual-mode doesn't yet support Shadow DOMs, maybe an issue should be created for that. I will revert back the changes for this function)

EDIT: I will actually revert changes for both scroll-to-element and set-caret-on-start since they don't seem to be directly related to hint-mode. The latter will keep generating errors when hintable elements are inside Shadow DOMs due to the changes applied to mode/hint.lisp. Not sure if I should fix that in this pull request or open a new issue and create a pull request to add support for Shadow DOMs in visual-mode.

EDIT: update scroll-to-element item

aartaka · 2023-06-06T13:34:34Z

I would also like to precise that this pull request was mostly inspired by #2821 in the beginning due to the similarity between Shadow DOM and IFrame elements, but then I had to change most of the code due to some changes that introduced bugs like the one I have mentioned here. One of the changes that I had to drop was adding multiple stylesheets such as in the commit from the pull request I linked earlier.

Right, new stylesheeds are not needed for Shadow DOM, because it inherits from the main frame already!

I also want to suggest moving all the hint elements into a single parent element outside the main DOM body, the current behavior is appending the hint overlay elements to the body which makes it very cluttered and we would have to remove them one by one which wouldn't be the case if we had all elements under the same custom nyxt node, this is the case for saka-key and LinkHints .

Yes, that'd be much cleaner. No hurry, though—this can be addressed separately.

aartaka · 2023-06-06T13:42:11Z

I have just tested the following interactive functions in dom.lisp, the checked elements are work as expected:
...

scroll-to-element (I didn't know how to test this function)

It's only used in heading scrolling, so testing it on the headings inside Shadow DOM seems the only testcase :P

It should also be used for highlight-selected-hint someday, I guess.

set-caret-on-start (Generates errors because visual-mode doesn't yet support Shadow DOMs, maybe an issue should be created for that. I will revert back the changes for this function)

Right, it'd need quite some refactoring to work with Shadow DOM. Let's figure it out with hints first and leave lots of TODOs for visual mode :P

heiwiper · 2023-06-06T14:23:42Z

Right, new stylesheeds are not needed for Shadow DOM, because it inherits from the main frame already!

Shadow DOM elements don't inherit stylesheet by default, the reason why I dropped that method is because I didn't see a point in drawing hint overlays in every Shadow DOM root element (this could be also the case for IFrames) when I can just append them to the main DOM and they would all be visible and would use the same stylesheet.

Right, it'd need quite some refactoring to work with Shadow DOM. Let's figure it out with hints first and leave lots of TODOs for visual mode :P

It doesn't seem complicated now that I implemented the the feature for hints, I will be working on it after I fix the hint scope probably.

aartaka

I like it, it's extremely elegant! I've left some minor notes regarding style here.

Regarding the non-working commands/macros/functions: proceed as you think best and ping me when you consider it ready :)

What won't hurt is a set of links you've tested the implementation on, so that I can see how it works.

source/mode/hint.lisp

source/parenscript-macro.lisp

aartaka · 2023-06-06T14:36:03Z

source/parenscript-macro.lisp

@@ -151,13 +166,15 @@
          (radius (parse-float (chain computed-style border-top-left-radius)))
          (rounded-border-offset (ceiling (* radius (- 1 (sin (/ pi 4))))))
          (offset (max coord-truncation-offset rounded-border-offset))
-          (el (chain document (element-from-point (+ (chain rect left) offset)
-                                                  (+ (chain rect top) offset)))))
+          (el (chain ,element (get-root-node ,element) (element-from-point (+ (chain rect left) offset)


Oh wow, it was there all this time?! That'd simplify a lot of code for IFrame indexing...

aartaka · 2023-06-06T14:38:45Z

source/parenscript-macro.lisp

@@ -167,3 +184,16 @@
          0)
       (= (chain window (get-computed-style ,element) "visibility")
          "hidden")))
+
+(export-always 'rqsa)


Actually, why define new variations for the existing Parenscript macros? Maybe simply replace qsa with this rqsa? Or is there something I'm missing?

I think that rqsa will perform worse for the cases that qsa currently covers.

Right. Here's an idea for optimization: test whether there are any Shadow DOM nodes in the page, and call a regular qsa when there's none.

Although I'm not sure how easy that'd be. Leaving it to your judgement, @heiwiper :)

EDIT: Typo.

Right. Here's an idea for optimization: test whether there are any Shadow DOM nodes in the page, and call a regular qsa when there's none.

This would not be possible as far as I know, Shadow DOMs don't seem to have a straighforward API to select them from the main DOM, one would need to check every element and call the shadowRoot function to test whether it a Shadow root or not. I did some tests on pages containing Shadow DOMs and others that do not contain this type of elements using performance.now(). It is definitely lacking in terms of speed, but it doesn't seem to have a signigicant impact but that is also relative to every webpage and how much content it contains.

There's actually a workaround that we could implement, we could check if an element is a shadow root in the function update-document-model in buffer.lisp and we add an attribute nyxt-shadow-root to the shadow root elements. we already do some checks in that function and we modify elements by adding nyxt-identifier. This would allow us to query for Shadow DOM elements using that attribute and we would be able to implement the optimization that you have suggested.

Yes, update-document-model (or the underlying nyxt/dom::get-document-body-json) is where I'd put it!

Should be resoved in the commits efa13d037, 5fb2654d1 and 5b0f340bf.

aadcg · 2023-06-07T16:55:32Z

Thanks @heiwiper!

Not much to add. I second the request for testing URLs.

heiwiper · 2023-06-07T17:01:15Z

Thanks a lot for the review! I will finish testing the remaining elements and review your comments as soon as possible.

In the meantime, you can test the feature on the link that was posted as an example on the issue linked to this PR (here https://developer.servicenow.com/dev.do).

heiwiper · 2023-06-12T00:47:24Z

scroll-to-element (I didn't know how to test this function)

It's only used in heading scrolling, so testing it on the headings inside Shadow DOM seems the only testcase :P

I'm getting a weird behaviour when trying to fix this, the following expression returns nil because the current heading element object doesn't seem to match the same object that is in the headings list, it is the same heading element but not the same object. Can you point me to the right direction to know why could this happen ?

aartaka · 2023-06-12T13:48:35Z

I'm getting a weird behaviour when trying to fix this, the following expression returns nil because the current heading element object doesn't seem to match the same object that is in the headings list, it is the same heading element but not the same object. Can you point me to the right direction to know why could this happen ?

My guess would be that the underlying document-model has gotten refreshed and thus the new object for the heading is not identical (as per eq) to the old one. What you can do instead is to get their text of some other value related to them that's unlikely to change. Like:

(position (element (current-heading buffer))
          headings
          :key (compose #'nyxt/dom:body #'element)
          :test #'equal)

heiwiper · 2023-06-17T19:54:25Z

My guess would be that the underlying document-model has gotten refreshed and thus the new object for the heading is not identical (as per eq) to the old one. What you can do instead is to get their text of some other value related to them that's unlikely to change. Like:
(position (element (current-heading buffer))
          headings
          :key (compose #'nyxt/dom:body #'element)
          :test #'equal)

This seems to solve the issue, please see the commit 4e751aa97.

The PR should be ready now for another review @aartaka. Please let me know if there's any other changes to add.

EDIT: Could please let me know if it is okay with you that I am force pushing changes ? I'm not sure if that's the best way to do it, but I try to make sure the PR commits are up to date with master so I cherry pick commits on top of master and force push.

aadcg · 2023-06-19T08:21:29Z

@heiwiper it's ok to force push to branches when a single person pushes commits to it, which it is the case here :)

aartaka

It looks perfect to me, I've nothing to add.

So let's

Rebase over master.
Squash commits (I believe squashing into one would be the most sensible thing).
Fixup commit messages (no backtics—plaintext commit messages are more accessible and don't depend on the Git/GUI parsing Markdown.)
Go through the PR checklist once again (documentation and changelogs, mostly).

aartaka · 2023-06-20T12:56:15Z

source/mode/hint.lisp

-        (i 0))
-    (dolist (element (nyxt/ps:qsa document "[nyxt-hintable]"))
-      (let ((hint (aref hints i)))
+        (nyxt-identifiers (ps:lisp (list 'quote nyxt-identifiers))))


Note that you can also (coerce nyxt-identifiers 'vector), although I'm not sure about the performance of such an approach 😛

Anyway, not critical.

Should I update the doctstring for the function nyxt:update-document-model to mention the new attribute nyxt-shadow-dom ?

Should I add docstring for already existing functions that I have modified ?

Note that you can also (coerce nyxt-identifiers 'vector), although I'm not sure about the performance of such an approach stuck_out_tongue

Anyway, not critical.

I tried to use the same code as the one used for hint binding

This commit contains the following changes: dom: Make Nyxt DOM Shadow DOM aware buffer: Add nyxt-identifier attribute to Shadow DOMs elements buffer: Add new attribute to identify Shadow Root elements The new attribute allows checking if the main DOM has any Shadow DOMs by using the browser API querySelector and similar functions, otherwise it would be necessary to loop through all elements and check if the property shadowRoot is not null. (See https://developer.mozilla.org/en-US/docs/Web/API/Element/shadowRoot) parenscript-macro: Add Shadow DOM aware recursive qsa like macro This macro allows querying elements behind Shadow DOMs which is normally not possible due to encapsulation. mode/hint: Search for elements recursively through Shadow DOMs parenscript-macro: Add Shadow DOM aware qs-nyxt-id like macro parenscript-macro: Make qs-nyxt-id JS friendly This commit is inspired by atlas-engineer@3fd4402 It is reused to avoid compilation warning concerning undefined symbol nyxt/mode/hint::nyxt-identifier. dom: Get Interactive elements recursively through Shadow DOM mode/hint: Add hint overlays by nyxt-identifier parenscript-macro: Handle overlapped Shadow DOM child elements mode/document: Fix heading scrolling for pages that use Shadow DOMs

heiwiper · 2023-07-16T12:12:29Z

Hello! I would like to follow up on this pull request to ask whether there's anything else to be done on my side.

jmercouris · 2023-07-16T16:03:31Z

Hey @heiwiper thanks for the ping, and I'm sorry for our negligence. I'll take a look at this on Monday. I believe it is ready. So I'll handle the conflicts and hopefully merge it as well!

aadcg · 2023-07-17T08:41:47Z

@heiwiper sorry for the delay. Looks ready to me. @jmercouris can I merge it?

jmercouris · 2023-07-17T08:53:07Z

Yes, feel free André!

aadcg · 2023-07-17T12:59:53Z

@heiwiper I've tried to test the feature but, in my understanding, I'm not sure whether it's working.

Steps:

Navigate to https://developer.servicenow.com/dev.do;
Issue follow-hint and see that no hints are drawn in the iframe containing the Cookies information.

Thanks.

heiwiper · 2023-07-17T14:01:40Z

I would like to first thank you for your getting back to this PR.
As for your feedback @aadcg. I won't be able to test this feature again until tomorrow. But I would like to clear some confusions before that. This PR addresses the missing hints of elements nested insides Shadow DOMs, the link you have tested should contain almost only this type of elements. So the expected behaviour is to get hints drawn over hintable elements of that website. IFrame nested elements are unrelated to this PR and I believe it was a WIP that was dropped according to this closed PR #2821 (see #1526).

aadcg · 2023-07-17T15:31:08Z

@heiwiper thanks! Indeed, my understanding of the changes is too basic, sorry about that. Could you provide an example of a URL that benefits from these changes?

heiwiper · 2023-07-17T15:40:03Z

Alright, I will elaborate on this tomorrow!

aartaka · 2023-07-17T21:25:43Z

Yes, IFrame hinting is dropped and is totally unrelated to this PR.

aadcg · 2023-07-18T14:25:46Z

@heiwiper I've merged your work via c1b9975. I've followed this full thread in the meantime and everything is now clear. Thanks again for your amazing work and persistence!

heiwiper force-pushed the shadow-dom-hinting branch from a7a6883 to acd7648 Compare June 4, 2023 23:22

aartaka approved these changes Jun 6, 2023

View reviewed changes

heiwiper force-pushed the shadow-dom-hinting branch from acd7648 to 30d8f4b Compare June 12, 2023 00:29

heiwiper force-pushed the shadow-dom-hinting branch from 30d8f4b to 4e751aa Compare June 17, 2023 17:18

heiwiper requested a review from aartaka June 17, 2023 19:54

heiwiper force-pushed the shadow-dom-hinting branch from 4e751aa to e9afd29 Compare June 18, 2023 21:48

heiwiper mentioned this pull request Jun 18, 2023

Visual mode doesn't work in pages containing Shadow DOMs #3029

Open

aartaka approved these changes Jun 20, 2023

View reviewed changes

heiwiper added 2 commits June 24, 2023 23:26

changelog: Mention hinting support for Shadow DOMs

b855533

heiwiper force-pushed the shadow-dom-hinting branch from e9afd29 to b855533 Compare June 24, 2023 22:27

aadcg closed this Jul 18, 2023

aadcg mentioned this pull request Jul 21, 2023

Hints do not appear for elements within a Shadow DOM #1828

Closed

aadcg mentioned this pull request Nov 14, 2023

hint-mode performance #3254

Closed

aadcg mentioned this pull request Nov 28, 2023

Hints are not visible on Bing Chat #3261

Closed

jmercouris mentioned this pull request Feb 29, 2024

Fix hint-mode #3345

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hinting support for Shadow DOM nested elements #3001

Add hinting support for Shadow DOM nested elements #3001

heiwiper commented Jun 4, 2023

heiwiper commented Jun 4, 2023 •

edited

aartaka commented Jun 6, 2023

aartaka commented Jun 6, 2023

heiwiper commented Jun 6, 2023

aartaka left a comment

aartaka Jun 6, 2023

aartaka Jun 6, 2023

aadcg Jun 7, 2023

aartaka Jun 8, 2023 •

edited

heiwiper Jun 11, 2023

aartaka Jun 13, 2023

heiwiper Jun 17, 2023

aadcg commented Jun 7, 2023

heiwiper commented Jun 7, 2023

heiwiper commented Jun 12, 2023

aartaka commented Jun 12, 2023

heiwiper commented Jun 17, 2023 •

edited

aadcg commented Jun 19, 2023 •

edited

aartaka left a comment

aartaka Jun 20, 2023

heiwiper Jun 24, 2023

heiwiper Jun 24, 2023

heiwiper commented Jul 16, 2023

jmercouris commented Jul 16, 2023

aadcg commented Jul 17, 2023

jmercouris commented Jul 17, 2023

aadcg commented Jul 17, 2023 •

edited

heiwiper commented Jul 17, 2023

aadcg commented Jul 17, 2023

heiwiper commented Jul 17, 2023

aartaka commented Jul 17, 2023 via email

aadcg commented Jul 18, 2023

Add hinting support for Shadow DOM nested elements #3001

Add hinting support for Shadow DOM nested elements #3001

Conversation

heiwiper commented Jun 4, 2023

Description

Discussion

Checklist:

heiwiper commented Jun 4, 2023 • edited

aartaka commented Jun 6, 2023

aartaka commented Jun 6, 2023

heiwiper commented Jun 6, 2023

aartaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aartaka Jun 8, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aadcg commented Jun 7, 2023

heiwiper commented Jun 7, 2023

heiwiper commented Jun 12, 2023

aartaka commented Jun 12, 2023

heiwiper commented Jun 17, 2023 • edited

aadcg commented Jun 19, 2023 • edited

aartaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heiwiper commented Jul 16, 2023

jmercouris commented Jul 16, 2023

aadcg commented Jul 17, 2023

jmercouris commented Jul 17, 2023

aadcg commented Jul 17, 2023 • edited

heiwiper commented Jul 17, 2023

aadcg commented Jul 17, 2023

heiwiper commented Jul 17, 2023

aartaka commented Jul 17, 2023 via email

aadcg commented Jul 18, 2023

heiwiper commented Jun 4, 2023 •

edited

aartaka Jun 8, 2023 •

edited

heiwiper commented Jun 17, 2023 •

edited

aadcg commented Jun 19, 2023 •

edited

aadcg commented Jul 17, 2023 •

edited