[#197] Canonicalize filepaths #230

aeqz · 2022-12-02T12:38:41Z

Description

Problem: The current usage of filepaths is error-prone and can be simplified.

Solution: Canonicalize filepaths at the boundaries, so their management will be safer and will simplify the codebase. This has the side effect of changing the program output regarding file paths, so we should also probably decide how these will be shown after the refactor.

Related issue(s)

Fixes #197

✅ Checklist for your Pull Request

Ideally a PR has all of the checkmarks set.

If something in this list is irrelevant to your PR, you should still set this
checkmark indicating that you are sure it is dealt with (be that by irrelevance).

Related changes (conditional)

Tests
- If I added new functionality, I added tests covering it.
- If I fixed a bug, I added a regression test to prevent the bug from
  silently reappearing again.
Documentation
- I checked whether I should update the docs and did so if necessary:
  - README
  - Haddock
Public contracts
- Any modifications of public contracts comply with the Evolution
  of Public Contracts policy.
- I added an entry to the changelog if my changes are visible to the users
  and
- provided a migration guide for breaking changes if possible

Stylistic guide (mandatory)

My commits comply with the policy used in Serokell.
My code complies with the style guide.

src/Xrefcheck/Core.hs

src/Xrefcheck/Scan.hs

src/Xrefcheck/Verify.hs

Martoon-00 · 2022-12-02T13:58:12Z

Cool, looking at your comments I like how you approach all this.

Gonna take a thorough look by Monday. Please switch to another ticket meanwhile, unless you have something else to do here before my review.

Martoon-00

Looks pretty good, better than what I expected to be done in this issue 👍

And your code generally looks well, cool.

I'm sorry for the delayed review, hope it wasn't a blocker for you.

src/Xrefcheck/Core.hs

src/Xrefcheck/Scanners/Markdown.hs

src/Xrefcheck/System.hs

src/Xrefcheck/Scan.hs

src/Xrefcheck/Verify.hs

src/Xrefcheck/System.hs

src/Xrefcheck/Core.hs

Martoon-00 · 2022-12-05T19:18:01Z

On the failing tests:

I believe that the switch from displaying absolute paths in most cases is very good.

However I wonder why in case of Windows (this CI is in GitHub actions) we get relative paths with /, not with \, despite everything is canonicalized. Maybe there is a quick answer to this? If not, no need to setup Windows locally.

Also, I find that in case of File not found error it may be beneficial to print an absolute path too (perhaps on a separate line), this way it may be simpler for the user to investigate the issue if it was not expected.

There are tests that seem broken now while they shouldn't, e.g. "24 Ignore file with broken xrefcheck annotation: config file".

And some diff seems to point to places that were invalid but now are valid, not sure how did this happen. Apparently, we had too many "Link targets a local file outside repository" errors, now it seems to work better.

Could you please regenerate the golden files and carefully commit only those changes where we had "Link targets a local file outside repository" illegally, if that does not take much effort? This should give us an opportunity to document the reasoning of this change (in the commit description) and should simplify sorting through the remaining failing tests.

aeqz · 2022-12-06T12:02:35Z

I have moved CanonicalPath related code under the Xrefcheck.System module, and now I will try to fix the issues related to path printing and unexpected golden test diffs.

aeqz · 2022-12-06T17:43:53Z

Now the Golden test expected outputs reflect how the program output has changed. I will investigate the couple of diffs that are unexpectedly not being reported as invalid.

aeqz · 2022-12-07T09:56:51Z

The golden test that is failing reports as "local file outside repository" a link that is not really outside the repository, but passes trough its root: the root directory is dir1 and the link, relative from dir1, is ../dir1/file.md.

After the refactor, its path is treated as file.md, so it does not report that it is outside the repository. If we want to preserve that behaviour, we should add an extra check that, before canonicalizing the file link path, follows its indirections and detects whether it passes through the root or not.

Martoon-00

I like the changes to the golden tests that were applied, looks neat.

A few more comments.

src/Xrefcheck/System.hs

tests/Test/Xrefcheck/LocalSpec.hs

src/Xrefcheck/Verify.hs

Martoon-00 · 2022-12-07T19:18:42Z

The golden test that is failing reports as "local file outside repository" a link that is not really outside the repository, but passes trough its root: the root directory is dir1 and the link, relative from dir1, is ../dir1/file.md.

After the refactor, its path is treated as file.md, so it does not report that it is outside the repository. If we want to preserve that behaviour, we should add an extra check that, before canonicalizing the file link path, follows its indirections and detects whether it passes through the root or not.

Interesting. So we got more flexible after your changes.

I think this may be helpful in case the user has auto-generated documentation, so let's try to preserve this new feature of yours.

But, canonicalizePath seems to poke the filesystem taking the prefixes of the given filepath, and I believe we must avoid touching anything outside of the repository: some users may care about files' permissions and what is being touched by third party programs. So let's make sure to add the new check in a way that excludes canonicalizePaths calls over links that even "temporarily" exit the repository.

aeqz · 2022-12-13T10:28:21Z

Another drawback that I have noticed with having file paths canonicalised is that, on case-insensitive systems, if the file example/dir/file.md exists, then exAMple/DIR/FILe.md is canonicalised as example/dir/file.md, so we cannot report it as invalid even if we wanted a case-sensitive setting such as in #211.

This is why I have not fixed the Golden tests yet: some case-related errors that are reported in CI are not reported in my computer.

src/Xrefcheck/System.hs

Martoon-00 · 2022-12-14T07:55:35Z

if the file example/dir/file.md exists, then exAMple/DIR/FILe.md is canonicalised as example/dir/file.md, so we cannot report it as invalid even if we wanted a case-sensitive setting such as in #211.

Oooh, that's a very good spot and a severe issue.

This is a completely different concern compared to #197, so could you please create a separate ticket for this? And under it investigate how GitHub and GitLab handle the casing of filepaths, is it similar to anchors handling or not.

Maybe we will close that new ticket with this PR too.

Now on replacing canonicalizePath: it's a serious take; on the one hand, it's easy to mess up with things here, on the other, given that we try to do things partially manually, we could try to go further and avoid using makeAbsolute.

Let's really go with the weakened canonicalization as you have it now, and please create another ticket for trying to get rid of makeAbsoule and work with filepaths in a pure way.

tests/Test/Xrefcheck/RedirectRequestsSpec.hs

src/Xrefcheck/Verify.hs

src/Xrefcheck/System.hs

tests/Test/Xrefcheck/CanonicalPathSpec.hs

src/Xrefcheck/RepoInfo.hs

src/Xrefcheck/Verify.hs

YuriRomanowski · 2022-12-15T18:51:18Z

src/Xrefcheck/System.hs

+-- | A FilePath that has been canonicalized. Should be created via
+-- 'canonicalizePath'. Currently, canonical paths have been made absolute,
+-- normalised regarding the running platform (e.g. Posix or Windows), with
+-- indirections syntractically expanded as much as possible and with no


Hm, what is "syntractically"? Is it a typo?

YuriRomanowski · 2022-12-15T20:14:02Z

src/Xrefcheck/System.hs

+    Nothing -> []
+
+-- | Get a relative 'FilePath' from the second given path (child) with
+-- repect to the first one (root).


YuriRomanowski · 2022-12-15T21:08:00Z

src/Xrefcheck/System.hs

+    prepare :: FilePath -> FilePath
+    prepare path = case dropWhile FP.isPathSeparator path of
+      "" -> "."
+      other -> other


IMO this name is confusing a bit, prepare for what? Wouldn't it be better to call it dropLeadingPathSeparators?

I will give it a proper name. It also handles specially the empty case, returning "." instead of "".

aeqz · 2022-12-16T11:45:25Z

src/Xrefcheck/Scan.hs

+--
+-- Within 'Gather', 'seFile' stores the 'FilePath' corresponding
+-- to the file in where the error was found.
+data ScanError (a :: ScanStage) = ScanError


Do you think that this 'Trees that grow' style fits in here?

Oh, I like your type families solution here, it makes particular sense. At your place, I would probably write it in the same way :)

However I think that two separate datatypes would work better here, and here's why: our core part serves as sort of API for potential scanners developers, and later there will be a lot of code that works with ScanError 'Parse.

With the current approach, we win at the core side (fewer number of datatypes) and lose at the parsers' side (one has to always initialize that seFile = ()), and the latter is more significant for the code amount and API-wise.

Would it be an option to use mkParseScanError instead of initialising seFile to ()? I can also give type aliases to ScanError 'Parse and ScanError 'Gather also.

Oh right, we already use a smart constructor there, so avoiding () each time won't be a problem.

Okay, let's go with it.

YuriRomanowski · 2022-12-16T16:33:24Z

src/Xrefcheck/Verify.hs

+          RIFileLocal -> checkRef rAnchor riRoot file ""
+          RIFileRelative -> do
+            let relativeToRoot = getPosixRelativeOrAbsoluteChild riRoot (takeDirectory file)
+                  </> toString rLink


Well, AFAIU, this is not always relative, from the function name. But even better it would be to call it shownPath, like inside the checkRef function, it seems that it doesn't really matter that it is relative.

Actually so, I agree here. Having relativeToRoot would make the code slightly more readable IMO, but here it is not necessarily relative.

YuriRomanowski · 2022-12-16T16:45:11Z

tests/Test/Xrefcheck/CanonicalPathSpec.hs

+    [ testGroup "Canonicalization"
+        [ testCase "Trailing separator" $ do
+            path <- canonicalizePath "./example/dir/"
+            getPosixRelativeOrAbsoluteChild current path @?= "example/dir",


Please format it like this:

exceptions = [ InvalidStatusCode , MissingContentHeader , InternalServerError ]

YuriRomanowski · 2022-12-16T16:47:07Z

I'm perfectly fine with this PR, titanic work! I approved it.

aeqz · 2022-12-16T18:01:05Z

I will wait also for @Martoon-00 's review. And then, get ready for conflicts in our currently open PRs 🙊

Martoon-00 · 2022-12-19T15:16:55Z

Oh, looks like hasIndirectionThroughParent is gone now, why so?

aeqz · 2022-12-19T15:33:02Z

Oh, looks like hasIndirectionThroughParent is gone now, why so?

I added hasIndirectionThroughParent, but then removed it because of the following comment:

my previous concern about poking files outside of the repository does not hold for makeAbsolute. So nevermind my previous comment.

Do we need it for something that I have missed? Or should I keep it just in case it may be useful in the future?

Martoon-00 · 2022-12-19T15:46:31Z

Do we need it for something that I have missed?

Ah, I didn't put my thought concrete.

My comment "about poking files outside of the repository" only affected where hasIndirectionThroughParent check had to be performed, that we shouldn't run canonicalizePath on paths that are yet unchecked on indirection.

But checking for indirections has to be there in either way, we cannot throw this check away altogether.

My subsequent comment mentioning virtual files meant that we probably want to check on indirections inside checkRef function, at the same position where this check was located before this PR.

aeqz · 2022-12-19T21:56:40Z

@Martoon-00 right! I misunderstood it. Now the Golden tests output shows the same behaviour as before the refactor regarding both virtual files and links that pass through the repository root.

Martoon-00 · 2022-12-20T11:48:34Z

	1 Warning
⚠️	Please, only use letters, digits and dashes in the branch name. Found: ["_"]

Generated by 🚫 Danger

Martoon-00 · 2022-12-20T13:42:26Z

Feel free to prettify the commits history, I'll make the final review pass then.

Problem: the danger checks were failing because it was configured to fetch only and partially the current PR branch. Solution: force the danger checks CI to get all the repository branches.

Martoon-00

Thank you for your titanic work!

I believe we achieved quite a lot, both from the perspective of code cleanness and in how much better the output now looks like.

I'm leaving one more comment about commits history prettification, and going to approve after that.

src/Xrefcheck/Scanners/Markdown.hs

Martoon-00 · 2022-12-22T14:45:35Z

src/Xrefcheck/Verify.hs

+          RIFileLocal -> checkRef rAnchor riRoot file ""
+          RIFileRelative -> do
+            let relativeToRoot = getPosixRelativeOrAbsoluteChild riRoot (takeDirectory file)
+                  </> toString rLink


Actually so, I agree here. Having relativeToRoot would make the code slightly more readable IMO, but here it is not necessarily relative.

Problem: our code contains a where statement with a 5-space indentation. Solution: indent it with 6 spaces instead.

Problem: the current usage of filepaths is error-prone and can be simplified. Solution: canonicalize filepaths at the boundaries, so their management will be safer and will simplify the codebase.

Martoon-00 · 2022-12-22T15:37:34Z

Hmhm, danger complains about Style tags.

But we can actually allow them. You can update danger/helpers.rb to allow both Chore and Style tags.

aeqz requested review from dcastro, Martoon-00, Sorokin-Anton and Sereja313 December 2, 2022 12:38

aeqz commented Dec 2, 2022

View reviewed changes

src/Xrefcheck/Core.hs Outdated Show resolved Hide resolved

aeqz commented Dec 2, 2022

View reviewed changes

src/Xrefcheck/Core.hs Outdated Show resolved Hide resolved

aeqz commented Dec 2, 2022

View reviewed changes

src/Xrefcheck/Scan.hs Show resolved Hide resolved

aeqz commented Dec 2, 2022

View reviewed changes

src/Xrefcheck/Verify.hs Outdated Show resolved Hide resolved

aeqz changed the title ~~Aeqz/#197 canonicalize filepaths~~ [#197] Canonicalize filepaths Dec 5, 2022

Martoon-00 reviewed Dec 5, 2022

View reviewed changes

aeqz force-pushed the aeqz/#197-canonicalize_filepaths branch from 429459e to 9bc47b2 Compare December 7, 2022 09:29

Martoon-00 reviewed Dec 7, 2022

View reviewed changes

aeqz requested a review from Martoon-00 December 12, 2022 11:13

Martoon-00 mentioned this pull request Dec 12, 2022

Optimize performance in case of local links #219

Open

aeqz force-pushed the aeqz/#197-canonicalize_filepaths branch from 9bc47b2 to 7380221 Compare December 13, 2022 09:38

aeqz requested a review from YuriRomanowski December 13, 2022 10:35

aeqz commented Dec 13, 2022

View reviewed changes

src/Xrefcheck/System.hs Outdated Show resolved Hide resolved

aeqz force-pushed the aeqz/#197-canonicalize_filepaths branch from 511e5d0 to dd17047 Compare December 13, 2022 13:48

Martoon-00 reviewed Dec 14, 2022

View reviewed changes

tests/Test/Xrefcheck/RedirectRequestsSpec.hs Show resolved Hide resolved

src/Xrefcheck/Verify.hs Outdated Show resolved Hide resolved

src/Xrefcheck/System.hs Show resolved Hide resolved

tests/Test/Xrefcheck/CanonicalPathSpec.hs Outdated Show resolved Hide resolved

aeqz commented Dec 14, 2022

View reviewed changes

src/Xrefcheck/RepoInfo.hs Outdated Show resolved Hide resolved

aeqz commented Dec 14, 2022

View reviewed changes

src/Xrefcheck/Verify.hs Outdated Show resolved Hide resolved

aeqz mentioned this pull request Dec 14, 2022

Make canonical paths relative to the repository root #239

Closed

YuriRomanowski reviewed Dec 15, 2022

View reviewed changes

aeqz commented Dec 16, 2022

View reviewed changes

aeqz requested review from Martoon-00 and YuriRomanowski December 16, 2022 13:25

YuriRomanowski reviewed Dec 16, 2022

View reviewed changes

YuriRomanowski approved these changes Dec 16, 2022

View reviewed changes

Martoon-00 mentioned this pull request Dec 19, 2022

Try to exploit newtypes over FilePath more to better separate semantics #249

Closed

aeqz force-pushed the aeqz/#197-canonicalize_filepaths branch from c095d6e to 742aa00 Compare December 20, 2022 11:08

[Chore] Fix CI danger checks

3c16b90

Problem: the danger checks were failing because it was configured to fetch only and partially the current PR branch. Solution: force the danger checks CI to get all the repository branches.

aeqz force-pushed the aeqz/#197-canonicalize_filepaths branch from 5309f71 to 1d3e863 Compare December 20, 2022 16:01

Martoon-00 reviewed Dec 22, 2022

View reviewed changes

aeqz added 2 commits December 22, 2022 16:29

[Style] Fix indentation

a13910d

Problem: our code contains a where statement with a 5-space indentation. Solution: indent it with 6 spaces instead.

[#197] Canonicalize filepaths

0886062

Problem: the current usage of filepaths is error-prone and can be simplified. Solution: canonicalize filepaths at the boundaries, so their management will be safer and will simplify the codebase.

aeqz force-pushed the aeqz/#197-canonicalize_filepaths branch from 1d3e863 to 0886062 Compare December 22, 2022 15:30

Martoon-00 approved these changes Dec 22, 2022

View reviewed changes

[Chore] Allow Style tags

978afec

aeqz merged commit 16041d0 into master Dec 22, 2022

aeqz deleted the aeqz/#197-canonicalize_filepaths branch December 22, 2022 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#197] Canonicalize filepaths #230

[#197] Canonicalize filepaths #230

aeqz commented Dec 2, 2022 •

edited

Martoon-00 commented Dec 2, 2022

Martoon-00 left a comment •

edited

Martoon-00 commented Dec 5, 2022

aeqz commented Dec 6, 2022

aeqz commented Dec 6, 2022

aeqz commented Dec 7, 2022

Martoon-00 left a comment

Martoon-00 commented Dec 7, 2022

aeqz commented Dec 13, 2022

Martoon-00 commented Dec 14, 2022

YuriRomanowski Dec 15, 2022

YuriRomanowski Dec 15, 2022

YuriRomanowski Dec 15, 2022 •

edited

aeqz Dec 16, 2022

aeqz Dec 16, 2022

Martoon-00 Dec 19, 2022 •

edited

aeqz Dec 20, 2022

Martoon-00 Dec 20, 2022

YuriRomanowski Dec 16, 2022

Martoon-00 Dec 22, 2022

YuriRomanowski Dec 16, 2022

YuriRomanowski commented Dec 16, 2022

aeqz commented Dec 16, 2022

Martoon-00 commented Dec 19, 2022

aeqz commented Dec 19, 2022

Martoon-00 commented Dec 19, 2022

aeqz commented Dec 19, 2022

Martoon-00 commented Dec 20, 2022 •

edited

Martoon-00 commented Dec 20, 2022

Martoon-00 left a comment

Martoon-00 Dec 22, 2022

Martoon-00 commented Dec 22, 2022

[#197] Canonicalize filepaths #230

[#197] Canonicalize filepaths #230

Conversation

aeqz commented Dec 2, 2022 • edited

Description

Related issue(s)

✅ Checklist for your Pull Request

Related changes (conditional)

Stylistic guide (mandatory)

Martoon-00 commented Dec 2, 2022

Martoon-00 left a comment • edited

Choose a reason for hiding this comment

Martoon-00 commented Dec 5, 2022

aeqz commented Dec 6, 2022

aeqz commented Dec 6, 2022

aeqz commented Dec 7, 2022

Martoon-00 left a comment

Choose a reason for hiding this comment

Martoon-00 commented Dec 7, 2022

aeqz commented Dec 13, 2022

Martoon-00 commented Dec 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YuriRomanowski Dec 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martoon-00 Dec 19, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YuriRomanowski commented Dec 16, 2022

aeqz commented Dec 16, 2022

Martoon-00 commented Dec 19, 2022

aeqz commented Dec 19, 2022

Martoon-00 commented Dec 19, 2022

aeqz commented Dec 19, 2022

Martoon-00 commented Dec 20, 2022 • edited

Martoon-00 commented Dec 20, 2022

Martoon-00 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martoon-00 commented Dec 22, 2022

aeqz commented Dec 2, 2022 •

edited

Martoon-00 left a comment •

edited

YuriRomanowski Dec 15, 2022 •

edited

Martoon-00 Dec 19, 2022 •

edited

Martoon-00 commented Dec 20, 2022 •

edited