Removing JavaScript Dependencies

[TimKam]

Relates to #10028.
Sphinx plans to remove jQuery and underscore JS dependencies in version 6.0.0 (i.e., eventually, there is plenty of time). This will probably break many extensions and themes and we need to try and find at least the most important ones.

TODOs:

• Remove 'internal' usage of dependencies: Drop JavaScript Frameworks #10028
• Fix dependency usage of all extensions hosted by https://github.com/sphinx-doc and https://github.com/sphinx-contrib/ in some way (either refactoring or including dependencies, but ideally refactoring).
  • https://github.com/sphinx-doc/sphinxcontrib-websupport/blob/master/sphinxcontrib/websupport/files/websupport.js
  • https://github.com/sphinx-contrib/cheeseshop/blob/master/sphinxcontrib/cheeseshop.py
  • https://github.com/sphinx-contrib/images (uses Lightbox, which relies on jQuery)
• Identify as many of the frequently used 3rd-party themes and extensions as possible and warn the maintainers.
  • https://github.com/readthedocs/sphinx_rtd_theme
  • https://github.com/ryan-roemer/sphinx-bootstrap-theme
  • https://github.com/readthedocs/sphinx-hoverxref
  • https://github.com/humitos/sphinx-version-warning
  • https://github.com/executablebooks/sphinx-thebe
  • https://github.com/useblocks/sphinx-needs
  • https://github.com/ulrobix/sphinxcontrib-contentui
  • https://github.com/schettino72/sphinx_press_theme
  • https://github.com/MrDogeBro/sphinx_rtd_dark_mode
  • https://github.com/jurasofish/sphinx-toggleprompt
  • https://github.com/BlueBrain/sphinx-bluebrain-theme
  • https://github.com/collective/sphinxcontrib-httpexample
  • https://github.com/jucacrispim/sphinx_pdj_theme
  • https://github.com/wagtail/sphinx_wagtail_theme
  • https://github.com/LinxiFan/Sphinx-theme/
  • https://github.com/typlog/sphinx-typlog-theme
  • https://github.com/useblocks/sphinx-data-viewer
  • https://github.com/ionelmc/sphinx-py3doc-enhanced-theme
• Remove dependencies

[TimKam]

Looping in some Read the Docs folks: @ericholscher & @nienn: Do you have any advice or can you provide any intel based on RTD usage? For example, we could check the most important extensions and themes, based on RTD usage stats. Thank you.

[AA-Turner]

From a quick look, the only usage in https://github.com/sphinx-doc I can see ¹ is https://github.com/sphinx-doc/sphinxcontrib-websupport/blob/master/sphinxcontrib/websupport/files/websupport.js

Popular/impactful themes -- I'd probably suggest looking at at least:

• https://github.com/python/python-docs-theme
• https://github.com/pydata/pydata-sphinx-theme
• https://github.com/pradyunsg/furo (not impacted Removing JavaScript Dependencies #10070 (comment))
• https://github.com/readthedocs/sphinx_rtd_theme
• https://github.com/bashtage/sphinx-material/
• https://github.com/executablebooks/sphinx-book-theme

Actual data would of course be more helpful!

A

Footnotes

1. This is the link for GitHub's new code search preview -- I also went through for underscore and the $ and $u symbols. https://cs.github.com/?scopeName=All+repos&scope=&q=org%3Asphinx-doc++jQuery ↩

[pradyunsg]

I can confirm that Furo doesn't depend on jQuery or UnderscoreJS.

[TimKam]

Thanks! I forgot that there is a separate org for many extensions: https://github.com/sphinx-contrib/ (I do not have access to the new GitHub code search feature, yet.)

[pradyunsg]

Doing a search on sphinx-contrib brings up references in findanything and images.

There's a total of 10 results, all in these two. Okay, there's one in zopeext but that's in an HTML file.

[TimKam]

I now have access to GitHub code search and found these two extensions that use jQuery:

• https://github.com/sphinx-contrib/cheeseshop
• https://github.com/sphinx-contrib/findanything
  I'd assume they are not widely used.

[ericholscher]

@TimKam Sadly we don't have great stats on extension usage across the platform. This is something we hope to figure out eventually, but not something we have currently.

[TimKam]

Thanks, @ericholscher. Let us know if there's anything we need to consider from RTD's perspective when removing jQuery and underscore. @tk0miya I think many templates will break if we don't announce this carefully. Should we add an info message when building or something similar?

[tk0miya]

I don't have good idea to let developers know the deprecation. But info or message warnings on build would not help it, I think. So it would be better to announce it on CHANGES file and sphinx-users group.

[AA-Turner]

An idea I had is (attempting to) patch the actual jQuery object itself in JS, and warn on any method access. I don't think it's great though, as it relies on users checking the JS console.

Agree with @tk0miya on adding notes to CHANGES etc -- my initial PR at #10028 does this, although maybe it could be called out more?

A

[Daltz333]

https://github.com/wpilibsuite provides several popular extensions and we are tracking this change. Not sure when we'll make the change, but we'll get to it.

[marxin]

I've noticed Search results are quite broken, one can see it also on the official website:

https://www.sphinx-doc.org/en/master/search.html?q=--keep-going

Search Results
Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories.

Search Results
Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories.

[object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object]

I see the section Search Results twice and the [object Object] seems to be a weird result.

What was displayed before:
https://sphinx--10240.org.readthedocs.build/en/10240/search.html?q=--keep-going

Search Results
Search finished, found 2 page(s) matching the search query.

sphinx-build.--keep-going (program option, in sphinx-build)
sphinx-build.--keep-going (program option, in sphinx-build)

Moreover, the following is also bad:
https://www.sphinx-doc.org/en/master/search.html?q=sphinx-build

...
sphinx-build.11 (program option, in sphinx-build)
sphinx-build.11 (program option, in sphinx-build)
sphinx-build.12 (program option, in sphinx-build)
sphinx-build.12 (program option, in sphinx-build)
sphinx-build.13 (program option, in sphinx-build)
sphinx-build.13 (program option, in sphinx-build)
sphinx-build.14 (program option, in sphinx-build)
sphinx-build.14 (program option, in sphinx-build)
sphinx-build.15 (program option, in sphinx-build)
sphinx-build.15 (program option, in sphinx-build)
...

[AA-Turner]

@marxin looking at the console output at least part of the problem is that read the docs' overwriting of Sphinx search isn't fixed yet -- does the problem persist on 'pure' Sphinx?

A

[marxin]

@marxin looking at the console output at least part of the problem is that read the docs' overwriting of Sphinx search isn't fixed yet -- does the problem persist on 'pure' Sphinx?

Dunno. Is there a bug report about the RTD issue, please?

[marxin]

And I have a project where it fails with master branch + #10153 fix:

https://splichal.eu/scripts/sphinx/demo/_build/html/search.html?q=-Wno-shift-overflow&check_keywords=yes&area=default

[stsewd]

@marxin I'm trying to fix all things related to search on latest sphinx and rtd, I'm tracking the progress at readthedocs/readthedocs.org#8871.

[AA-Turner]

And I have a project where it fails...

https://splichal.eu/scripts/sphinx/demo/_build/html/search.html?q=-Wno-shift-overflow

In that project, https://splichal.eu/scripts/sphinx/demo/_build/html/searchindex.js doesn't have wno-shift-overflow composited in "terms" so the regex match won't work. Do you have a link to the same project on version 4.y.z?

Sorry to ask for more things, I'm just trying to find where the error is -- the stemmer, the index, searchtools.js itself. etc.

A

[marxin]

In that project, https://splichal.eu/scripts/sphinx/demo/_build/html/searchindex.js doesn't have wno-shift-overflow composited in "terms" so the regex match won't work. Do you have a link to the same project on version 4.y.z?

Sure, there's a version built with 4.x branch:
https://splichal.eu/tmp/_build/html/search.html?q=-Wno-shift-overflow

[AA-Turner]

@marxin does #10247 fix the issue?

A

[marxin]

@marxin does #10247 fix the issue?

It's better, but still not ideal:
https://splichal.eu/tmp/_build2/html/search.html?q=-Wno-shift-overflow

Note 4.x branch lives here:
https://splichal.eu/tmp/_build/html/search.html?q=-Wno-shift-overflow

[marxin]

Note you can use the following Demo project for testing:
https://github.com/marxin/sphinx-demo

[marxin]

@AA-Turner Can you please take a look at the attached demo project and investigate what's wrong? Thanks.

[humitos]

@humitos

I'd like to do a __contains together with a __regex for the name, so I can do sphinxcontrib-*, but I think that's not possible. I've tried executing some raw queries using PostgreSQL JSON fields syntax without success sob

I went ahead and did this manually in a dirty way to, at least, have these numbers 😄

1. get the list of all repositories from sphinx-contrib GH organization

$ gh api --paginate -H "Accept: application/vnd.github+json" /orgs/sphinx-contrib/repos | jq -r '.[].name' | sort > sphinx-crontib.repositories

2. query the database for each of them prefixing sphinxcontrib-
3. results: https://gist.github.com/humitos/489b9159b21faffd9ce1d0d2c91ec5ff

Note this data belongs to projects that were built at least once starting in mid-May (when we started collecting this data) and the repository name does not always match the PyPI package name.

I hope this data is useful to make better decisions.

[humitos]

Hi all! I've been working on analyzing some data we have been collecting on Read the Docs after building the "telemetry" database. That data helps us to answer questions like "What are the most used Sphinx extensions in a period?". I wrote a blog about this at https://blog.readthedocs.com/knowing-more-about-ourselves/ and added this particular plot on it.

I think that plot/data could be useful when prioritizing the work required to remove JavaScript dependency and on what extensions work first. Let me know if you find this useful and if you are interested in a particular question that can be answered with the data we have now. Thanks!

[AA-Turner]

Thanks @humitos! Would it be possible to have the extension popularity data in a tabular format? (Eg just pasted in a table here.)

We can then look through the most popular and open issues or identify JS deps etc.

Thanks,
Adam

[AA-Turner]

Ping here @humitos!

A

[pradyunsg]

FWIW, I believe @humitos is currently travelling to attend a conference. He might respond during the trip but I figured it can’t hurt to mention this to set expectations. :)

[AA-Turner]

Ahh, thanks Pradyun. This isn't urgent, of course.

A

[humitos]

I'm at DjangoConEU and that's why I haven't replied yet. However, I should have mentioned this before, tho 🙂

I plan to come back to this issue next week and upload the data requested with more information about the filters used to get it.

[humitos]

Hey @AA-Turner! I'm back to work and I executed the query for the following statement:

Number of projects on Read the Docs with a build between 2022-08-01 and 2022-09-01 that installed Python packages (including transitive dependencies) matching "^(sphinx|sphinxcontrib)-" regex on their name and group by extension's name.

Raw query

SELECT
  COUNT(DISTINCT data->'project'->'id') AS projects,
  item->>'name' AS name
FROM 
  "telemetry_builddata",
  LATERAL jsonb_path_query(data, '$.packages.pip.all ? (@.name like_regex "^(sphinxcontrib|sphinx)-")') item
WHERE
  created BETWEEN '2022-08-01' AND '2022-09-01'
GROUP BY item->>'name'
ORDER BY projects DESC;

and these are the results: sphinx-extensions.csv

Let me know if this is the correct query you were expecting and if it's helpful for you to make better decisions.

[AA-Turner]

Thank you @humitos, this is really usefull!

Do you track package dependency data at all? I.e. package X has a transitive/direct dependency on Sphinx core but might not have "sphinx" in its name.

If you don't, I could probably build this data myself but I imagine the CSV file of all dependencies downloaded into RTD is substantially larger!

A

[AA-Turner]

The other (top-down) direction would be if you track the contents of the "extensions" or "html_theme" keys in conf.py, which might be easier!

A

[humitos]

Unfortunately, we don't track the package dependency nor the contents of extensions or html_theme. I have another query that removes the list of extensions from https://github.com/sphinx-doc/sphinx/blob/5.x/pyproject.toml#L59 but I thought I shouldn't filter here to keep the query simple and easy to understand.

[humitos]

Unfortunately, we don't track the package dependency nor the contents of extensions or html_theme

The PR I created to track these got merged already. Hopefully, we will have some data about the extensions starting next week 👍🏼

[AA-Turner]

From @humitos's first list, the following depend on jQuery implicitly. I have checked the entire list and I believe all the others are fine.

projects	name
13026	sphinx-rtd-theme
5106	sphinxcontrib-websupport
130	sphinx-bootstrap-theme
85	sphinx-hoverxref
76	sphinx-version-warning
54	sphinx-thebe
41	sphinxcontrib-needs
37	sphinxcontrib-contentui
32	sphinxcontrib-images
31	sphinx-press-theme
25	sphinx-rtd-dark-mode
19	sphinx-toggleprompt
15	sphinx-bluebrain-theme
12	sphinxcontrib-httpexample
8	sphinx-pdj-theme
8	sphinx-wagtail-theme
6	sphinx-theme
6	sphinx-typlog-theme
6	sphinx-data-viewer
5	sphinx-py3doc-enhanced-theme

A

[AA-Turner]

Sphinx 6.0.0 has been released.

A