follow-hint issue

[hajovonta]

I'm having a problem using follow-hint on Jenkins webpage (heavy with JS): shows in minibuffer
Warning: Error on separate thread: The value "[{"type":"link","hint":"AB","href":"" ...

I'm not sure which sites have this problem, Jenkins does. Also, not sure what info would be needed (can reproduce easily)
Clicking on the links is ok.

[aartaka]

What exactly Jenkins webpage do you mean? Can you provide a link, please? https://www.jenkins.io/ seems to hint just fine.

[hajovonta]

It's a company internal CI webpage. You wouldn't be able to access it... but I'm assuming it's not company-specific.
The webpages worked fine a few months ago in Nyxt (I used an older version)

[aartaka]

Hmmm... Would yo be able to name the version that worked? Was it one of the pre-releases or 1.4/5?

[hajovonta]

it was one of the pre-releases, it hadn't have prompt buffer yet.

[aartaka]

That's enough of a pointer, thanks! I'll check what changed from pre-prompt-buffer pre-releases then :)

[aartaka]

OK, it seems we haven't had much changes in element hints since before prompt buffer. Most recent changes influencing hints are:

• GHT.
• Prompt-buffer.
• Introduction of image-, focusable-, and clickable-hints, and more slots to existing hints (input-hint, textarea-hint).

The most plausible version is that the renderer gets overwhelmed with how much clickable/focusable elements there are (and there tend to be lots of these in JS-reliant code) and simply errors out.

There are two ways to fix this:

• Try to make things more error-proof with better serialization and object collection.
• Wait for Plump DOM #1365 xD It's quite conservative and will probably work in much the same way as the pre-prompt-buffer things did. This doesn't solve the problem of renderer being overwhelmed by us, but it probably won't return in the case element-hints.

[dev-zero]

I am not sure whether I have the same issue since I don't get a warning. But on some webpages some links are not "enumerated" and it seems to be mostly some kind java-script-based ones. Example (for all books the download link does not get a selector):

[aartaka]

Yes, this is a different issue, @dev-zero. See #1400.

[hajovonta]

Hi @aartaka, thanks for the investigation!
I can live without this for some time now, but wanted to report so that you have feedback.

[Vivien-lelouette]

I can confirm the issue on my company's Jenkins as well. So this is probably happening on other Web sites for others as well.

I am using the Ubuntu 20 deb package, I'll probably try to build from master at some point.

Just wanted to confirm the issue is not on @hajovonta end.

Keep up the good work Atlas :).

[hajovonta]

Hi, just a quick update: issue still present in 2.1.0-6-g5bdc58fe
no pressure, I wait for whatever solution comes first. Other websites work fine...

[aartaka]

Thanks for the update :)

[aartaka]

@hajovonta, can you try building from master and trying hinting again? We've overhauled the hinting, so it should be much more stable now.

[hajovonta]

Hi @aartaka ,
I've just tried it, and things got actually got worse: issuing follow-hints (M-g M-g) makes Nyxt crash and close. Can I somehow collect logs about this?

[hajovonta]

Nyxt version 2.1.1-215-g29e95a3b

[aartaka]

You can run nyxt -v to get debug output in the shell.

[hajovonta]

here is the output of the session:

0712 17:08:03 /home λ nyxt -v
Arguments parsed: (VERBOSE T URLS NIL) and NIL
[17:08:06] nyxt start.lisp (listen-or-query-socket) -
Listening to socket "/run/user/1000/nyxt/nyxt.socket".
Nyxt version 2.1.1-216-g27b0529d
[17:08:06] nyxt start.lisp (load-lisp unsafe-load) -
Loading Lisp file "/home/user/.config/nyxt/auto-config.lisp".
[17:08:07] nyxt start.lisp (load-lisp unsafe-load) -
Lisp file "/home/user/.config/nyxt/init.lisp" does not exist.
[17:08:07] nyxt renderer-gi-gtk.lisp (ffi-initialize gtk-browser) -
Initializing GI-GTK Interface
[17:08:08] nyxt base-mode.lisp (base-mode form-fun-20) -
BASE-MODE enabled.
[17:08:08] nyxt/web-mode web-mode.lisp (web-mode form-fun-32) -
WEB-MODE enabled.
[17:08:08] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:08] nyxt/help-mode help-mode.lisp (help-mode form-fun-16) -
HELP-MODE enabled.
[17:08:08] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:08] nyxt/prompt-buffer-mode prompt-buffer-mode.lisp (prompt-buffer-mode form-fun-19) -
PROMPT-BUFFER-MODE enabled.
[17:08:18] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
NYXT::KEY-STRING: "return" NYXT::KEYCODE: 36 CHARACTER: #\Return
NYXT::KEYVAL-NAME: "Return"
[17:08:18] nyxt input.lisp (dispatch-input-event) -
Found key binding "return" [#36] to RETURN-SELECTION
[17:08:18] nyxt message.lisp (echo form-fun-6) -
Loading history of 1030 URLs from "/home/user/.local/share/nyxt/history/default.lisp".
[17:08:18] nyxt message.lisp (echo form-fun-6) -
Loading history of 1030 URLs from "/home/user/.local/share/nyxt/history/default.lisp".
[17:08:18] nyxt history.lisp (restore-history-buffers with-recursive-lock-thunk) -
Restoring 4 buffers from history.
[17:08:18] nyxt base-mode.lisp (base-mode form-fun-20) -
BASE-MODE enabled.
[17:08:18] nyxt/web-mode web-mode.lisp (web-mode form-fun-32) -
WEB-MODE enabled.
[17:08:18] nyxt/certificate-exception-mode certificate-exception-mode.lisp (certificate-exception-mode form-fun-26) -
CERTIFICATE-EXCEPTION-MODE enabled.
[17:08:18] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:18] nyxt base-mode.lisp (base-mode form-fun-20) -
BASE-MODE enabled.
[17:08:18] nyxt/web-mode web-mode.lisp (web-mode form-fun-32) -
WEB-MODE enabled.
[17:08:18] nyxt/certificate-exception-mode certificate-exception-mode.lisp (certificate-exception-mode form-fun-26) -
CERTIFICATE-EXCEPTION-MODE enabled.
[17:08:18] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:18] nyxt base-mode.lisp (base-mode form-fun-20) -
BASE-MODE enabled.
[17:08:18] nyxt/web-mode web-mode.lisp (web-mode form-fun-32) -
WEB-MODE enabled.
[17:08:18] nyxt/certificate-exception-mode certificate-exception-mode.lisp (certificate-exception-mode form-fun-26) -
CERTIFICATE-EXCEPTION-MODE enabled.
[17:08:18] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:18] nyxt base-mode.lisp (base-mode form-fun-20) -
BASE-MODE enabled.
[17:08:18] nyxt/web-mode web-mode.lisp (web-mode form-fun-32) -
WEB-MODE enabled.
[17:08:18] nyxt/certificate-exception-mode certificate-exception-mode.lisp (certificate-exception-mode form-fun-26) -
CERTIFICATE-EXCEPTION-MODE enabled.
[17:08:18] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:19] nyxt/web-mode web-mode.lisp (add-url-to-history) -
Notify URL https://jenkins.url.com/ for buffer #<USER-WEB-BUFFER {100915D123}> with load status LOADING
[17:08:19] nyxt data-storage.lisp (store :around data-profile async-data-path worker) -
Skipping 1 unnecessary #<HISTORY-DATA-PATH {1009210E73}> store ops
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
WebKitWebProcess: ../../src/xcb_io.c:260: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
[17:08:20] nyxt browser.lisp (preprocess-request) -
Forwarding https://jenkins.url.com/ for buffer #<USER-WEB-BUFFER {100915D123}>
[17:08:20] nyxt renderer-gtk.lisp (on-signal-decide-policy gtk-buffer) -
Forward to #<USER-WEB-BUFFER {100915D123}>'s renderer (no request-resource-hook handlers).
[17:08:20] nyxt message.lisp (echo form-fun-6) -
Loading "https://jenkins.url.com/".
[17:08:20] nyxt browser.lisp (preprocess-request) -
Forwarding https://jenkins.url.com/ for buffer #<USER-WEB-BUFFER {100915D123}>
[17:08:20] nyxt renderer-gtk.lisp (on-signal-decide-policy gtk-buffer) -
Forward to #<USER-WEB-BUFFER {100915D123}>'s renderer (no request-resource-hook handlers).
[17:08:22] nyxt/web-mode web-mode.lisp (add-url-to-history) -
Notify URL https://jenkins.url.com/ for buffer #<USER-WEB-BUFFER {100915D123}> with load status LOADING
[17:08:24] nyxt browser.lisp (preprocess-request) -
Forwarding about:blank for buffer #<USER-WEB-BUFFER {100915D123}>
[17:08:24] nyxt renderer-gtk.lisp (on-signal-decide-policy gtk-buffer) -
Forward to #<USER-WEB-BUFFER {100915D123}>'s renderer (no request-resource-hook handlers).
[17:08:24] nyxt/web-mode web-mode.lisp (add-url-to-history) -
Notify URL https://jenkins.url.com/ for buffer #<USER-WEB-BUFFER {100915D123}> with load status FINISHED
[17:08:24] nyxt message.lisp (echo form-fun-6) -
Finished loading "https://jenkins.url.com/".
[17:08:33] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
NYXT::KEY-STRING: NIL NYXT::KEYCODE: 64 CHARACTER: #\Nul
NYXT::KEYVAL-NAME: "Alt_L"
[17:08:33] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
NYXT::KEY-STRING: "g" NYXT::KEYCODE: 42 CHARACTER: #\g NYXT::KEYVAL-NAME: "g"
NYXT::MODIFIERS: ("meta")
[17:08:33] nyxt message.lisp (echo form-fun-6) - Pressed keys: M-g
[17:08:33] nyxt input.lisp (dispatch-input-event) -
Prefix binding "M-g" [M-#42]
[17:08:33] nyxt renderer-gtk.lisp (on-signal-key-press-event gtk-window) -
NYXT::KEY-STRING: "g" NYXT::KEYCODE: 42 CHARACTER: #\g NYXT::KEYVAL-NAME: "g"
NYXT::MODIFIERS: ("meta")
[17:08:33] nyxt input.lisp (dispatch-input-event) -
Found key binding "M-g M-g" [M-#42 M-#42] to FOLLOW-HINT
[17:08:33] nyxt/emacs-mode emacs-mode.lisp (emacs-mode form-fun-30) -
EMACS-MODE enabled.
[17:08:33] nyxt/prompt-buffer-mode prompt-buffer-mode.lisp (prompt-buffer-mode form-fun-19) -
PROMPT-BUFFER-MODE enabled.
[17:08:33] nyxt/element-hint-mode element-hint-mode.lisp (element-hint-mode form-fun-16) -
ELEMENT-HINT-MODE enabled.
Unhandled TYPE-ERROR in thread #<SB-THREAD:THREAD "Prompter source init thread" RUNNING
{1006B73963}>:
The value
#[
is not of type
LIST
when binding LIST

Backtrace for: #<SB-THREAD:THREAD "Prompter source init thread" RUNNING {1006B73963}>
0: (SB-KERNEL:%ASSOC-TEST # # #)
1: ((LABELS NYXT/DOM::JSON-TO-PLUMP :IN NYXT/DOM:NAMED-JSON-PARSE) #[ #<PLUMP-DOM:ELEMENT html {10072EF933}>)
2: ((FLET SB-IMPL::F :IN SB-IMPL::VECTOR-MAP-INTO/SIMPLE-VECTOR) #[)
3: (SB-KERNEL:%MAP-FOR-EFFECT-ARITY-1 #<FUNCTION (FLET SB-IMPL::F :IN SB-IMPL::VECTOR-MAP-INTO/SIMPLE-VECTOR) {7F7E031FE4FB}> #<(SIMPLE-ARRAY CHARACTER (213772)) [{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-extensions-available": "true", "data-unit-test": "false", "data-imagesurl": ... {100711000F}>)
4: (SB-KERNEL:%MAP NIL #<FUNCTION (FLET SB-IMPL::F :IN SB-IMPL::VECTOR-MAP-INTO/SIMPLE-VECTOR) {7F7E031FE4FB}> #<(SIMPLE-ARRAY CHARACTER (213772)) [{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-extensions-available": "true", "data-unit-test": "false", "data-imagesurl": ... {100711000F}>) [more]
5: (SB-IMPL::VECTOR-MAP-INTO/SIMPLE-VECTOR #(0 0 0 0 0 0 0 0 0 0 0 0 ...) 0 213772 #<FUNCTION (LAMBDA (&REST ALEXANDRIA::MORE) :IN ALEXANDRIA:RCURRY) {10072F01AB}> (#<(SIMPLE-ARRAY CHARACTER (213772)) [{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-extensions-available": "true", "data-unit-test": "false", "data-imagesurl": ... {100711000F}>))
6: ((LABELS NYXT/DOM::JSON-TO-PLUMP :IN NYXT/DOM:NAMED-JSON-PARSE) ((:NAME . "HTML") (:ATTRIBUTES (:CLASS . "")) (:CHILDREN . #<(SIMPLE-ARRAY CHARACTER (213772)) [{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-extensions-available": "true", "data-unit-test": "false", "data-imagesurl": ... {100711000F}>)) #<PLUMP-DOM:ROOT {1007266EB3}>)
7: (NYXT/DOM:NAMED-JSON-PARSE #<(SIMPLE-ARRAY CHARACTER (248608)) {"name":"HTML","attributes":{"class":""},"children":"[{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-exte... {1006EF800F}>)
8: (NYXT/WEB-MODE:UPDATE-DOCUMENT-MODEL #<NYXT/WEB-MODE:USER-WEB-MODE {1009B0D283}>)
9: (NYXT/WEB-MODE::ADD-ELEMENT-HINTS :SELECTOR "a, button, input, textarea, details, select, img:not([alt=""])")
10: ((LAMBDA NIL :IN INITIALIZE-INSTANCE))
11: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))
12: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
13: ((FLET "WITHOUT-INTERRUPTS-BODY-11" :IN SB-THREAD::RUN))
14: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
15: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::RUN))
16: (SB-THREAD::RUN)
17: ("foreign function: call_into_lisp")
18: ("foreign function: funcall1")

unhandled condition in --disable-debugger mode, quitting
[17:08:34] nyxt message.lisp (echo-warning form-fun-6) -
Warning: Error: Joining thread failed: thread #<THREAD "Nyxt renderer thread" ABORTED {10085934E3}> did not return normally.

[aartaka]

Can you try again with bbfc156? I think that some parsing corner case could've been causing this crash.

[hajovonta]

Unfortunately the issue is still there (the error output seems to be the same. I've tried it with Nyxt version 2.1.1-223-gd33257d4 (currently master)

[hajovonta]

Still there with Nyxt version 2.1.1-365-gc61c17dd
Reproduction of the problem is very easy: just have to press M-g M-g at the problematic page. For a moment, "follow-hint" appears in the prompt buffer then Nyxt closes immediately.

[hajovonta]

@aartaka
I've looked at the stacktrace and the code (dom.lisp#131, (defun named-json-parse (json) ...))
There is a comment:
JSON should have the format like what `get-document-body-json' produces:

• A nested hierarchy of objects (with only one root object), where
  • Every object has a 'name' (usually a tag name or '#text'/'#comment').
  • Some objects can have 'attributes' (a string->string dictionary).
  • Some objects have a subarray ('children') of objects working by these three
    rules."

And I think the problem here is that this object actually does have a children subarray, but it starts with '[' like this:
7: (NYXT/DOM:NAMED-JSON-PARSE #<(SIMPLE-ARRAY CHARACTER (230356)) {"name":"HTML","attributes":{"class":""},"children":"[{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-exte... {1006CE000F}>)

And this somehow ruins the parsing, as in the next step:
6: ((LABELS NYXT/DOM::JSON-TO-PLUMP :IN NYXT/DOM:NAMED-JSON-PARSE) ((:NAME . "HTML") (:ATTRIBUTES (:CLASS . "")) (:CHILDREN . #<(SIMPLE-ARRAY CHARACTER (197914)) [{"name": "HEAD", "attributes": {"resurl": "/static/d6a831fe", "data-rooturl": "", "data-resurl": "/static/d6a831fe", "data-extensions-available": "true", "data-unit-test": "false", "data-imagesurl": ... {1006FD800F}>)) #<PLUMP-DOM:ROOT {1006EFEEB3}>)

then later receives this:
1: ((LABELS NYXT/DOM::JSON-TO-PLUMP :IN NYXT/DOM:NAMED-JSON-PARSE) #[ #<PLUMP-DOM:ELEMENT html {10070EF8E3}>)

My thought is that a new case might need to be added to the (let ((element (cond ... list for these pages. Do you need the source for the page, I can send it over?

[aartaka]

Children sub-array is parsed alright on all the other pages. There should be something else, some mangled markup.

My thought is that a new case might need to be added to the (let ((element (cond ... list for these pages. Do you need the source for the page, I can send it over?

Yes, that could help!

[hajovonta]

I would like to send it as a html attachment, can I send it to your github email?

[aartaka]

I would like to send it as a html attachment, can I send it to your github email?

Sure :)

[aartaka]

Hmmm... I cannot reproduce it with the HTML you provided 0_o Does this HTML file break Nyxt when you open it and try to hint it?

Regarding the suspicious '[': the '[' is the standart JSON delimiter for array start. What looks wrong in the stacktrace is that it has "children" : "[...]" instead of "children" : [...]. Array is somewhy enclosed into a string. I'm not sure how that's even possible, because our parsing code doesn't do that.

I wish there were more websites this happens on, so that it's more testable :/

[hajovonta]

Neither can I... I'll try to make another one, it's possible that I messed this one up.

[hajovonta]

Hi @aartaka,
I sent another sample, please give it a try.

[aartaka]

@hajovonta, I've tested your example and still cannot reproduce. However, I've looked at the stacktrace again and inferred that there was an incorrect type handling -- we were taking unparsed string for a parsed list, thus error about particulat characters not being list elements.

8f30ed0 should adress that. Can you test it?

[hajovonta]

Yes, confirming the problem is now gone! Thanks for your efforts!

[aartaka]

I'm glad it works :D