Add documentation build and links checking in CI. #1145

uggla · 2020-08-20T21:20:37Z

Sanity check:

Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Code changes

(Delete or ignore this section for documentation changes)

Are you doing the PR on the next branch?

If the change is a new feature or adding to/changing an existing one:

Have you created/updated the relevant documentation page(s)?

Explanations:

@Keats, you told me in a comment that it was something you wanted to add to the CI. I did it as I could handle it and let you manage more complex stuffs.

Add zola build and zola check to ensure documentation can be built and does not contain invalid links.
Fix links in the current documentation that were broken.
Maybe I would have to open this PR on the master branch ? Let me know if you would rather this PR on the master branch.

uggla · 2020-08-21T14:07:42Z

Hum, seems like this is suffering from #1056. I think I'm gonna add a couple of retries to try to avoid link checking issue. Maybe not the best solution but better than nothing.

Keats · 2020-08-26T18:11:36Z

azure-pipelines.yml

+          - script: |
+              cd docs && ..\target\debug\zola check
+            condition: eq( variables['Agent.OS'], 'Windows_NT' )
+            displayName: Check documentation links windows


That's no good, it would block the release of Zola if there is a dead link. I would be tempted to move zola check to GH actions actually.

Hello @Keats, in fact that was the intention. 😄
To not pass the CI process if for any reason the documentation was broken or of boken links was introduced (Like the ones I corrected). Also it would check this on a regular basis and avoid documentation to deviate and keep it completely updated.
I like to put all checks in the CI, but if you think it annoying I can remove them from this PR.

It's a good instinct but the link checker is not robust yet so that could cause me to comment out these lines in anger if I want to do a release and it's failing for whatever reason

So I comment out the link check part.
I introduced nb_threads_per_cpu to configure the link_checker number of threads. As reducing the number of threads looks more stable at least on AzureDevOps. Maybe because there is some kind of limitation to traffic and especially number of simultaneous connections outgoing from AzureDevOps to avoid DDOS.

Keats · 2020-08-26T18:13:07Z

docs/config.toml

+# skip_anchor_prefixes = [
+#     "https://caniuse.com/",
+# ]
+


what's the reason for that change?

In fact I did a configuration documentation copy / paste.

[link_checker] # Skip link checking for external URLs that start with these prefixes skip_prefixes = [ "http://127.0.0.1", ] # Skip anchor checking for external URLs that start with these prefixes # skip_anchor_prefixes = [ # "https://caniuse.com/", # ]

I wanted to avoid broken links errors with http://127.0.0.1 (there are a several errors in the documentation), but so far I don't have to use the "anchors skip part". So I commented it out.

- Ensure documentation can be built. - Check documentation links are valid.

- Try to see if it is more stable with less thread especially on systems with few ressources.

- nb_threads_per_cpu define the number of thread per cpu the link_checker will use to check links. Example : if your system is 2 cpus and nb_threads_per_cpu = 6 you will get 6 * 2 = 12 threads to check links. - nb_threads_per_cpu default to 4.

Keats

Overall I like the build on CI but not sure about the rest. We can definitely lower the number of threads for the link checker though, in a more static way for now.

Keats · 2020-09-01T10:01:04Z

azure-pipelines.yml

+          - script: |
+              cd docs && ..\target\debug\zola build
+            condition: eq( variables['Agent.OS'], 'Windows_NT' )
+            displayName: Check documentation build windows


I like that!

Keats · 2020-09-01T10:05:23Z

components/config/src/config/link_checker.rs

@@ -7,10 +7,15 @@ pub struct LinkChecker {
    pub skip_prefixes: Vec<String>,
    /// Skip anchor checking for these URL prefixes
    pub skip_anchor_prefixes: Vec<String>,
+    pub nb_threads_per_cpu: usize,


I don't think this should be configurable. Let's keep it at 4 for now and we'll see for configuration later on.

So I come back to the static version

Keats · 2020-09-01T10:05:45Z

components/site/src/link_checking.rs

+        "Number of cpus: {}, using {} threads",
+        num_cpus::get(),
+        num_cpus::get() * nb_threads_per_cpu
+    );


we don't want to print that

I have removed it.

Keats · 2020-09-01T10:07:32Z

docs/content/themes/codinfox-zola/index.md


 This theme places content first by tucking away navigation in a hidden drawer.

-* Built for [Zola](https://www.getzola.com)
+* Built for [Zola](https://www.getzola.org)


Those are generated by the https://github.com/getzola/themes repo, fixing it here will not prevent more bad permalinks on the next update.

This reverts commit afccef5.

This reverts commit b7ad0f4.

This reverts commit c0f3d3e.

This reverts commit 0e6e7fc.

uggla force-pushed the feature/add_doc_test branch 3 times, most recently from 45ee2dc to b2b19f1 Compare August 21, 2020 13:14

Keats requested changes Aug 26, 2020

View reviewed changes

uggla added 8 commits August 31, 2020 23:26

Add CI documentation checks

40690d2

- Ensure documentation can be built. - Check documentation links are valid.

Fix failed links

4b3a527

Fix more links into recent docs

d931b26

Reduce the thread pool size for link checking

edd78e0

- Try to see if it is more stable with less thread especially on systems with few ressources.

Set thread pool size according to number of cpus

be06bc6

Display number of cpu

6404bc6

Reduce number of cpu and improve message

2107a0e

uggla force-pushed the feature/add_doc_test branch from f92886d to 0e6e7fc Compare August 31, 2020 21:28

uggla added 3 commits August 31, 2020 23:35

Update documentation about nb_threads_per_cpu

c0f3d3e

Comment link checking part

a6d9101

Fix wrong call of Linkchecker in tests

b7ad0f4

uggla changed the title ~~WIP:Add documentation build and links checking in CI.~~ Add documentation build and links checking in CI. Aug 31, 2020

Fix conf in config::tests::link_checker_nb_threads_per_cpu

afccef5

uggla requested a review from Keats August 31, 2020 22:19

Keats reviewed Sep 1, 2020

View reviewed changes

uggla added 6 commits September 2, 2020 22:32

Revert "Fix conf in config::tests::link_checker_nb_threads_per_cpu"

d1ba6c9

This reverts commit afccef5.

Revert "Fix wrong call of Linkchecker in tests"

3d4368b

This reverts commit b7ad0f4.

Revert "Update documentation about nb_threads_per_cpu"

5818b27

This reverts commit c0f3d3e.

Revert "Allow to pass link_checker nb_threads_per_cpu"

43b6a65

This reverts commit 0e6e7fc.

Remove message about cpu and thread used

1fdbc84

Fix comment about default threads

5aefeab

uggla requested a review from Keats September 2, 2020 21:43

Keats closed this Sep 4, 2020

Keats reopened this Sep 4, 2020

Keats force-pushed the next branch from ff1ee3d to cc3e4f9 Compare December 14, 2020 19:45

Keats closed this Jan 9, 2021

Keats deleted the branch getzola:next January 9, 2021 13:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation build and links checking in CI. #1145

Add documentation build and links checking in CI. #1145

uggla commented Aug 20, 2020 •

edited

Loading

uggla commented Aug 21, 2020

Keats Aug 26, 2020

uggla Aug 26, 2020

Keats Aug 27, 2020

uggla Aug 31, 2020 •

edited

Loading

Keats Aug 26, 2020

uggla Aug 26, 2020

Keats left a comment

Keats Sep 1, 2020

Keats Sep 1, 2020

uggla Sep 2, 2020

Keats Sep 1, 2020

uggla Sep 2, 2020

Keats Sep 1, 2020

Add documentation build and links checking in CI. #1145

Add documentation build and links checking in CI. #1145

Conversation

uggla commented Aug 20, 2020 • edited Loading

Code changes

uggla commented Aug 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uggla Aug 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Keats left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uggla commented Aug 20, 2020 •

edited

Loading

uggla Aug 31, 2020 •

edited

Loading