Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests] linkcheck: bind each test HTTP server to a unique port per-testcase. #12126

Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
35e3406
[tests] Add 'atomos' atomic datatype library
jayaddison Mar 17, 2024
1d4f565
[tests] linkcheck: bind each test HTTP server to a unique port per-te…
jayaddison Mar 18, 2024
345dde3
[tests] Fixup: move 'atomos' dependency to the correct extras-set.
jayaddison Mar 18, 2024
0a91f69
[linting] Ignore ruff v0.3.2 nested-context-manager suggestion due to…
jayaddison Mar 18, 2024
7ffc7e0
[linting] Use 'ruff --fix tests' to resolve formatting concerns.
jayaddison Mar 18, 2024
0aabff8
[linting] Resolve mypy type-check concern.
jayaddison Mar 18, 2024
81f3d44
[tests] Remove 'filelock' dependency.
jayaddison Mar 18, 2024
065d5b6
[tests] Nitpick: continue to begin from port 7777 (not 7778) when bin…
jayaddison Mar 18, 2024
d3da4fb
[tests] Cleanup: remove no-longer-used test server lockfile constant.
jayaddison Mar 18, 2024
4029674
Add CHANGES.rst entry
jayaddison Mar 18, 2024
73bf0e1
[tests] Remove use of config.html_baseurl; restore unambiguous source…
jayaddison Mar 18, 2024
acb40e4
[compatibility] Revert special-case handling of remote image URLs.
jayaddison Mar 18, 2024
9aec0e7
[linting] Apply ruff-suggested fixups.
jayaddison Mar 18, 2024
d21a1e2
[codestyle] More concise code using f-strings / elimination of variab…
jayaddison Mar 18, 2024
aae9f9e
[tests] Fixup: uncomment assertion statement.
jayaddison Mar 18, 2024
c5db9c7
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 18, 2024
a781a9a
[tests] Refactor: consolidate 'with' statements in linkcheck builder …
jayaddison Mar 18, 2024
8af07d1
Revert "[linting] Ignore ruff v0.3.2 nested-context-manager suggestio…
jayaddison Mar 18, 2024
9f44e38
[linting] Resolve flake8 errors due to unused 'port' variables.
jayaddison Mar 18, 2024
ceb7452
[tests] linkcheck: permit serve_sources context-manager to be provide…
jayaddison Mar 19, 2024
5b3ad00
[tests] linkcheck: use 'linkcheck-process-uri' event handler as a nea…
jayaddison Mar 19, 2024
162a96e
[linting] Fixups for linkcheck builder tests.
jayaddison Mar 19, 2024
41ecd55
[linting] Fixup for ruff-expected import order in linkcheck builder t…
jayaddison Mar 19, 2024
5d10a5a
[tests] linkcheck: minimize the overall diff by renaming the 'server_…
jayaddison Mar 19, 2024
bf145ea
[tests] Fixup: add missing 'app' parameter to 'http_server' function …
jayaddison Mar 19, 2024
4c35d50
[tests] Cleanup: remove debug 'print' statement.
jayaddison Mar 19, 2024
c7fdefc
[linting] Fixup for ruff-expected import styling.
jayaddison Mar 19, 2024
2b2430b
[tests] Fixup: write the index.rst page for the test_raw_node unit te…
jayaddison Mar 19, 2024
b453386
[nitpick] Align start of url option token vertically under parent rST…
jayaddison Mar 20, 2024
fbdeee2
[nitpick] Artifically introduce an f-string, to make indentation easi…
jayaddison Mar 20, 2024
bc82164
[refactor] Perform hyperlink rewrite and app-connection in a separate…
jayaddison Mar 20, 2024
7e6ec1a
[refactor] Return mapped network location (service address and port) …
jayaddison Mar 20, 2024
2439f1f
[refactor] Use str.format technique instead of f-strings to maintain …
jayaddison Mar 20, 2024
00dc3a1
Revert "[tests] Cleanup: remove no-longer-used test server lockfile c…
jayaddison Mar 20, 2024
fea86e6
Revert "[tests] Remove 'filelock' dependency."
jayaddison Mar 20, 2024
24afabb
[tests] Implement a per-port lockfile strategy for test HTTP servers"…
jayaddison Mar 20, 2024
9897bb5
[CI/CD] implement readiness check before yielding local-http(s) test …
jayaddison Mar 8, 2024
028f023
[linting] Fixup for ruff-expected import sort ordering
jayaddison Mar 20, 2024
dd1448a
[linting] Type-hinting fixups
jayaddison Mar 20, 2024
b75fc28
[linting] Fixup for COM819 (trailing comma) style error
jayaddison Mar 20, 2024
190bb86
[tests] testroots: increase the linkcheck builder timeout for the tes…
jayaddison Mar 20, 2024
2890d9b
[tests] utils: fixup for incorrectly-named variables in helper function.
jayaddison Mar 20, 2024
87d6647
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 20, 2024
a4e483f
[tests] linkcheck: provide a string-format kwarg named netloc for tes…
jayaddison Mar 20, 2024
61a5af5
[tests] utils: bind test HTTP server to port 0 (arbitrary unused port).
jayaddison Mar 21, 2024
dc45135
[cleanup] utils: remove inner-scope function; function is not stateful.
jayaddison Mar 21, 2024
c5c7cc5
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 21, 2024
101c735
[linting] test utils: restore end-of-file newline.
jayaddison Mar 21, 2024
83f7300
[fixup] Resolve merge-conflict resolution mistake (daemon argument sh…
jayaddison Mar 21, 2024
65dc815
Revert "[tests] testroots: increase the linkcheck builder timeout for…
jayaddison Mar 21, 2024
29bb8da
[cleanup] utils: remove comment.
jayaddison Mar 21, 2024
2f49d8c
[linting] test utils: reformat using ruff.
jayaddison Mar 21, 2024
abb2678
[cleanup] Remove no-longer-used lockfile paths from .gitignore file.
jayaddison Mar 21, 2024
91bd2de
Update CHANGES.rst entry
jayaddison Mar 21, 2024
251a072
[tests] utils: disconnect url-rewrite event handler afer usage.
jayaddison Mar 21, 2024
bbcb75d
[tests] utils: add explanatory docstring re: port selection mechanism.
jayaddison Mar 21, 2024
927747c
[tests] linkcheck: move some config elements back into the pytest mar…
jayaddison Mar 21, 2024
cd7dddc
[tests] safety: escape service hostname:port used within regular expr…
jayaddison Mar 21, 2024
d77bf7b
[refactor] Extract a serve_html context-manager combinator.
jayaddison Mar 21, 2024
5934a20
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 21, 2024
7498a6f
[refactor] Migrate remaining http_server callsites to use serve_html.
jayaddison Mar 21, 2024
8d734ae
[refactor] Rename webserver function and the name used to store its r…
jayaddison Mar 21, 2024
3fe53a8
[refactor] Don't return a value from the internal hyperlink-rewriter …
jayaddison Mar 21, 2024
8c759b6
[tests] linkcheck: add explanatory comment to URI-rewriter event hand…
jayaddison Mar 21, 2024
9d25084
[tests] linkcheck: replace incomplete typehint with runtime typecheck.
jayaddison Mar 21, 2024
d708130
[refactor] tests: use simpler names within the hyperlink-rewriting code.
jayaddison Mar 21, 2024
238ce86
[refactor] tests: nitpick: the webserver hostname-and-port is not a URI.
jayaddison Mar 21, 2024
0bb4911
Partially revert "[refactor] Rename webserver function and the name u…
jayaddison Mar 21, 2024
24db9c0
[tests] utils: return an HTTPServer object instead of a port number.
jayaddison Mar 21, 2024
60b363d
[refactor] tests: rename test webserver location variable for brevity.
jayaddison Mar 21, 2024
30f1f9b
[tests] test HTTP servers: stick with localhost as hostname for all t…
jayaddison Mar 21, 2024
90af0b1
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 22, 2024
7e4c55b
[docs] tests: add docstring to 'serve_html' utility function.
jayaddison Mar 22, 2024
c5a46cd
[refactor] Rectification: the function named 'serve_html' does not se…
jayaddison Mar 22, 2024
f48c629
[refactor] Relocate 'serve_application' and 'rewrite_hyperlinks' help…
jayaddison Mar 22, 2024
c9a6a2c
[typehinting] tests: add typehints for URL rewriting and application-…
jayaddison Mar 22, 2024
c4b0259
[typehinting] tests: use an Iterator-based typehint instead of Genera…
jayaddison Mar 22, 2024
3d0614f
[refactor] linkcheck tests: improve naming of test server address var…
jayaddison Mar 22, 2024
cf30498
[cleanup] Add explanatory .gitignore comment about test-server.lock r…
jayaddison Mar 22, 2024
801df71
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 23, 2024
b996477
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
jayaddison Mar 24, 2024
e9bd3a2
[changelog] Retain lockfile in .gitignore to prevent accidental commi…
jayaddison Mar 25, 2024
60792ad
[tests] utils: allow http_server.port to be specified per-instance.
jayaddison Mar 25, 2024
504545d
[utils] Fix param name in docstring, add missing call kwarg.
jayaddison Mar 25, 2024
84ef1fb
[typing] add typehint-ignore directives to config settings that use d…
jayaddison Mar 25, 2024
391cfed
[tests] utils: make HTTP server thread port a keyword-only parameter.
jayaddison Mar 25, 2024
fa9695a
[tests] utils: update HTTP server thread class docstring.
jayaddison Mar 25, 2024
c60c2ce
Improve phrasing/clarity of CHANGES.rst entry
jayaddison Mar 25, 2024
529603e
Merge branch 'master' into issue-12122/linkcheck-tests-httpserver-por…
picnixz Apr 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions .ruff.toml
Original file line number Diff line number Diff line change
Expand Up @@ -410,6 +410,9 @@ select = [
"tests/test_util/test_util_typing.py" = ["UP006", "UP035"]
"tests/test_util/typing_test_data.py" = ["FA100", "UP006", "UP035"]

# linkcheck builder tests use the contextmanager variable in subsequent 'with' blocks
"tests/test_builders/test_build_linkcheck.py" = ["SIM117"]
picnixz marked this conversation as resolved.
Show resolved Hide resolved

"utils/*" = [
"T201", # whitelist ``print`` for stdout messages
"ANN", # utilities don't need annotations
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ lint = [
"pytest>=6.0",
]
test = [
"atomos>=0.3.1",
"pytest>=6.0",
"html5lib",
"cython>=3.0",
Expand Down
10 changes: 7 additions & 3 deletions sphinx/builders/linkcheck.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from queue import PriorityQueue, Queue
from threading import Thread
from typing import TYPE_CHECKING, NamedTuple, cast
from urllib.parse import unquote, urlparse, urlsplit, urlunparse
from urllib.parse import unquote, urljoin, urlparse, urlsplit, urlunparse

from docutils import nodes
from requests.exceptions import ConnectionError, HTTPError, SSLError, TooManyRedirects
Expand Down Expand Up @@ -176,23 +176,27 @@ def run(self, **kwargs: Any) -> None:
builder = cast(CheckExternalLinksBuilder, self.app.builder)
hyperlinks = builder.hyperlinks
docname = self.env.docname
base_uri = builder.config.html_baseurl or None

# reference nodes
for refnode in self.document.findall(nodes.reference):
if 'refuri' in refnode:
uri = refnode['refuri']
uri = urljoin(base_uri, uri)
_add_uri(self.app, uri, refnode, hyperlinks, docname)

# image nodes
for imgnode in self.document.findall(nodes.image):
uri = imgnode['candidates'].get('?')
if uri and '://' in uri:
if uri and '://' in uri or base_uri:
uri = urljoin(base_uri, uri)
_add_uri(self.app, uri, imgnode, hyperlinks, docname)

# raw nodes
for rawnode in self.document.findall(nodes.raw):
uri = rawnode.get('source')
if uri and '://' in uri:
if uri and '://' in uri or base_uri:
uri = urljoin(base_uri, uri)
_add_uri(self.app, uri, rawnode, hyperlinks, docname)


Expand Down
3 changes: 3 additions & 0 deletions sphinx/environment/collectors/asset.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ def process_doc(self, app: Sphinx, doctree: nodes.document) -> None:
if imguri.find('://') != -1:
candidates['?'] = imguri
continue
if imguri.startswith('<') and imguri.endswith('>'):
candidates['?'] = imguri[1:-1]
continue

if imguri.endswith(os.extsep + '*'):
# Update `node['uri']` to a relative path from srcdir
Expand Down
14 changes: 7 additions & 7 deletions tests/roots/test-linkcheck-anchors-ignore-for-url/index.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
* `Example valid url, no anchor <http://localhost:7777/valid>`_
* `Example valid url, valid anchor <http://localhost:7777/valid#valid-anchor>`_
* `Example valid url, invalid anchor <http://localhost:7777/valid#invalid-anchor>`_
* `Example ignored url, no anchor <http://localhost:7777/ignored>`_
* `Example ignored url, invalid anchor <http://localhost:7777/ignored#invalid-anchor>`_
* `Example invalid url, no anchor <http://localhost:7777/invalid>`_
* `Example invalid url, invalid anchor <http://localhost:7777/invalid#anchor>`_
* `Example valid url, no anchor </valid>`_
* `Example valid url, valid anchor </valid#valid-anchor>`_
* `Example valid url, invalid anchor </valid#invalid-anchor>`_
* `Example ignored url, no anchor </ignored>`_
* `Example ignored url, invalid anchor </ignored#invalid-anchor>`_
* `Example invalid url, no anchor </invalid>`_
* `Example invalid url, invalid anchor </invalid#anchor>`_
2 changes: 1 addition & 1 deletion tests/roots/test-linkcheck-localserver-anchor/index.rst
Original file line number Diff line number Diff line change
@@ -1 +1 @@
`local server <http://localhost:7777/#anchor>`_
`local server </#anchor>`_
2 changes: 1 addition & 1 deletion tests/roots/test-linkcheck-localserver-https/index.rst
Original file line number Diff line number Diff line change
@@ -1 +1 @@
`HTTPS server <https://localhost:7777/>`_
`HTTPS server </>`_
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
`local server1 <http://localhost:7777/path1>`_
`local server1 </path1>`_

`local server2 <http://localhost:7777/path2>`_
`local server2 </path2>`_
2 changes: 1 addition & 1 deletion tests/roots/test-linkcheck-localserver/index.rst
Original file line number Diff line number Diff line change
@@ -1 +1 @@
`local server <http://localhost:7777/>`_
`local server </>`_
18 changes: 9 additions & 9 deletions tests/roots/test-linkcheck/links.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
Some additional anchors to exercise ignore code

* `Valid url <http://localhost:7777/>`_
* `Bar anchor invalid (trailing slash) <http://localhost:7777/#!bar>`_
* `Bar anchor invalid <http://localhost:7777#!bar>`_ tests that default ignore anchor of #! does not need to be prefixed with /
* `Top anchor invalid <http://localhost:7777/#top>`_
* `'does-not-exist' anchor invalid <http://localhost:7777#does-not-exist>`_
* `Valid local file <conf.py>`_
* `Valid url </>`_
* `Bar anchor invalid (trailing slash) </#!bar>`_
* `Bar anchor invalid <#!bar>`_ tests that default ignore anchor of #! does not need to be prefixed with /
* `Top anchor invalid </#top>`_
* `'does-not-exist' anchor invalid <#does-not-exist>`_
* `Valid local file <file:conf.py>`_
* `Invalid local file <path/to/notfound>`_

.. image:: http://localhost:7777/image.png
.. figure:: http://localhost:7777/image2.png
.. image:: <image.png>
.. figure:: <image2.png>

* `Valid anchored url <http://localhost:7777/anchor.html#found>`_
* `Valid anchored url <anchor.html#found>`_