[INFRA] PDF version of spec: fix handling of internal links #915

sappelhoff · 2021-10-28T20:36:56Z

fixes #852
replaces #855 (thereby closes #855)
related: #596 (but this PR does not really work towards that yet)

I think this is a nice bunch of fixes / enhancements. The one thing that would "wrap this up" would be support for removing "reference style" internal links from the PDF build. (i.e., links like [I am text][some-ref], where "some-ref" is declared at the bottom of the doc). But I think that's a rabbit hole, so I wanted to push this PR as long as it's reasonably clean.

fixes / enhancements so far

one typo in the spec
better docs about pdf building dependencies
regexp that removes internal links
- better docs
- ~~roadmap of what's needed (TODO, maybe another PR, don't know)~~
- simplify regexp, also make it catch more cases (probably solving the LGTM error, cc @DimitriPapadopoulos )
better logging during pdf table fixing when errors are encountered
fix table detection: previously, filename templates like [_part-<mag|phase|real|imag>] would be detected as table (due to numerous pipe chars)
fix weird markdown syntax in pre GH changelog MD doc
PDF building job now runs independently of linkchecker (+artifacts-redirector updated)

to do

fix for reference style links

better/alternative solution

only remove internal links that cannot be "fixed" instead, see: Support internal links in the PDF version #596 (comment) (not to be done here)

nice to have

a "check" going through all "reference style links" asserting that no single ref has been "overloaded" with two links across all documents
CI step that checks that links are not defined over multiple lines, like below:

this is an example of a link [that goes to
google](https://google.com)

That should be:

this is an example of a link
[that goes to google](https://google.com)

To make it easier to parse links

pdf_build_src/process_markdowns.py

sappelhoff · 2021-10-30T11:58:45Z

.circleci/config.yml

-      # Ensure that build_docs_pdf always runs last, so that we can use the CircleCI API link for the "latest" artifact
-      # https://circleci.com/api/v1.1/project/github/bids-standard/bids-specification/latest/artifacts/0/bids-spec.pdf?branch=master
-      - build_docs_pdf:
-          requires:
-            - build_docs
-            - linkchecker
-            - github-changelog-generator
-            - remark
-            - Changelog-bot


this is no longer needed (and hasn't been needed for a long time actually), because we use https://github.com/larsoner/circleci-artifacts-redirector-action

It's good we finally change this, because now we can see whether PDF build passes when linkchecker doesn't ... and linkchecker is arguably the test that fails most often, which often prevented us from seeing the PDF check

sappelhoff · 2021-10-30T15:30:41Z

Release_Protocol.md

+Minor revisions may be made using GitHub's
+[suggestion feature](https://help.github.com/en/articles/incorporating-feedback-in-your-pull-request).


I reformatted several such "multiline links" to be on a single line. this is a style choice that makes it easier to parse the links ... and it's also nice on the eye (the latter just IMHO). I added a CI check for this, so this style can now be enforced.

I think the advantage to this was to avoid a line break in GitHub's rendering of the document. It doesn't matter much to me, though.

sappelhoff · 2021-10-30T15:32:51Z

pdf_build_src/check_pandoc_log.py

+    msg = "\n\nFound duplicate link references. Please make them unique.\n"
+    for log_dict in duplicate_link_refs:
+        msg += "\n" + json.dumps(log_dict, indent=4)
+    raise RuntimeError(msg)


strictly speaking, having duplicate source links across different MKDocs .md documents is not an issue, because for HTML, these are rendered as separate pages.

It does become a problem for the PDF build though, where we first concatenate all files, so duplication results in partially wrong links.

I think it's reasonable to ask people to take a unique reference for a link reference. This short script will enforce this in CI, making use of the pandoc log, which detects such violations.

sappelhoff · 2021-10-30T15:33:55Z

src/pregh-changes.md

+-   Added support for describing events with Hierarchical Event Descriptors: [4.3 Task events](04-modality-specific-files/05-task-events.md).
+-   Added `VolumeTiming` and `AcquisitionDuration` fields: [4.1 Task (including resting state) imaging data](04-modality-specific-files/01-magnetic-resonance-imaging-data.md#task-including-resting-state-imaging-data).


This style change from [[.....]] to : [...] is also for convenience of link parsing.

sappelhoff

I left a couple of comments to make it easier for reviewers to see what I meant with some changes.

This PR is ready to be merged from my side.

tsalo

I can't speak to the code, but the PDF artifact looks good to me, as do the markdown changes.

Remi-Gau

This looks good to me. Also looks like a lot of work. Thanks @sappelhoff

sappelhoff added 9 commits October 27, 2021 14:26

add docs

a6e1c99

add docs about dependencies

813890c

fix logging of error tables

228d21b

fix false detection of filename templates as tables

aa36fba

change inconvenient [[ syntax in pre GH changelog

e0f52c0

correct typo

91cd5c7

try new regexp

686cb24

use groups instead of group in regexp

66377c4

add comments

b9ce472

sappelhoff commented Oct 28, 2021

View reviewed changes

pdf_build_src/process_markdowns.py Outdated Show resolved Hide resolved

sappelhoff added 11 commits October 28, 2021 23:37

clarify ref style links of form [some-ref][]

b5538ad

add draft for ref style link removal

6fe1502

eliminate pesky multiline links according to regex: \[[^\]]+$

245c962

fix case where link starts on newline

ea65419

fix string link

6d0b2c1

some docs

472273f

add docs on regexps

41f19df

add check for multiline markdown links

753753c

rename circleci redirect GH action

96e4bd3

update circleci artifact redirection

1ad28a3

run build_docs_pdf anytime, should work with new circleci redirector

1a9383b

sappelhoff commented Oct 30, 2021

View reviewed changes

sappelhoff added 3 commits October 30, 2021 13:59

add id to ci step

be67bd5

make pandoc produce a log

d5cdb7c

add check against duplicate link references

32ce8e6

sappelhoff commented Oct 30, 2021

View reviewed changes

sappelhoff requested review from DimitriPapadopoulos and effigies October 30, 2021 15:34

sappelhoff requested a review from yarikoptic October 30, 2021 15:34

sappelhoff commented Oct 30, 2021

View reviewed changes

sappelhoff added the formatting Aesthetics and formatting of the spec label Oct 30, 2021

sappelhoff requested review from Remi-Gau and tsalo November 5, 2021 14:11

sappelhoff added this to the 1.7.0 milestone Nov 9, 2021

tsalo approved these changes Nov 15, 2021

View reviewed changes

Remi-Gau approved these changes Nov 15, 2021

View reviewed changes

sappelhoff merged commit 41704e8 into bids-standard:master Nov 15, 2021

sappelhoff deleted the fix/pdf/internal_links branch November 15, 2021 16:03

bids-maintenance added a commit that referenced this pull request Nov 15, 2021

[DOC] Auto-generate changelog entry for PR #915

30c61b7

effigies added the exclude-from-changelog This item will not feature in the automatically generated changelog label Jul 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[INFRA] PDF version of spec: fix handling of internal links #915

[INFRA] PDF version of spec: fix handling of internal links #915

sappelhoff commented Oct 28, 2021 •

edited

sappelhoff Oct 30, 2021

sappelhoff Oct 30, 2021

effigies Nov 9, 2021

sappelhoff Oct 30, 2021

sappelhoff Oct 30, 2021

sappelhoff left a comment

tsalo left a comment

Remi-Gau left a comment

		Minor revisions may be made using GitHub's
		[suggestion feature](https://help.github.com/en/articles/incorporating-feedback-in-your-pull-request).

		- Added support for describing events with Hierarchical Event Descriptors: [4.3 Task events](04-modality-specific-files/05-task-events.md).
		- Added `VolumeTiming` and `AcquisitionDuration` fields: [4.1 Task (including resting state) imaging data](04-modality-specific-files/01-magnetic-resonance-imaging-data.md#task-including-resting-state-imaging-data).

[INFRA] PDF version of spec: fix handling of internal links #915

[INFRA] PDF version of spec: fix handling of internal links #915

Conversation

sappelhoff commented Oct 28, 2021 • edited

sappelhoff Oct 30, 2021

Choose a reason for hiding this comment

sappelhoff Oct 30, 2021

Choose a reason for hiding this comment

effigies Nov 9, 2021

Choose a reason for hiding this comment

sappelhoff Oct 30, 2021

Choose a reason for hiding this comment

sappelhoff Oct 30, 2021

Choose a reason for hiding this comment

sappelhoff left a comment

Choose a reason for hiding this comment

tsalo left a comment

Choose a reason for hiding this comment

Remi-Gau left a comment

Choose a reason for hiding this comment

sappelhoff commented Oct 28, 2021 •

edited