Ticket3252 port web directory.remaining.1

[meejah]

fixes:ticket:3252

Port the "directory" functionality from nevow to twisted.web.template

There are a lot of lines of changes here, but there are a bunch of "mechanical" changes that required some re-indenting and search-replace changes -- these are "tags" (HTML built up in code) stuff, changing from "context" to "request" objects and a bunch of stuff switching regular-expression use in tests to BeautifulSoup

---

This change is 

[meejah]

Arg, the perils of comparing strings ... (to repeat the circle-ci failures, use the same PYTHONHASHSEED)

[codecov-io]

Codecov Report

Merging #672 into master will increase coverage by 0.57%.
The diff coverage is 96.96%.

@@            Coverage Diff             @@
##           master     #672      +/-   ##
==========================================
+ Coverage   89.66%   90.24%   +0.57%     
==========================================
  Files         164      164              
  Lines       29015    29004      -11     
  Branches     4131     4131              
==========================================
+ Hits        26017    26174     +157     
+ Misses       2234     2095     -139     
+ Partials      764      735      -29

Impacted Files	Coverage Δ
src/allmydata/web/filenode.py	94.09% <100%> (-0.15%)	⬇️
src/allmydata/web/common.py	89.48% <100%> (-1.18%)	⬇️
src/allmydata/scripts/tahoe_backup.py	88.51% <100%> (+8.27%)	⬆️
src/allmydata/web/root.py	87.78% <50%> (-1.18%)	⬇️
src/allmydata/web/directory.py	92.25% <98.35%> (-0.31%)	⬇️
src/allmydata/web/status.py	84.61% <0%> (+0.22%)	⬆️
src/allmydata/mutable/servermap.py	94.22% <0%> (+0.48%)	⬆️
src/allmydata/scripts/runner.py	75.53% <0%> (+0.71%)	⬆️
src/allmydata/frontends/magic_folder.py	88.19% <0%> (+0.81%)	⬆️
... and 15 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d3474a...fdd8f52. Read the comment docs.

[exarkun]

Thanks. Various comments inline, as usual.

[meejah]

Okay, minus the Hypothesis thing I've made changes for all your comments. Thanks!

[exarkun]

Thanks. I think the changes generally look good. There are some comments looks like they may have been missed. I'm confused at how github is presenting them. I don't see them on this page at all... Oh they're hidden behind a "14 hidden conversations" button for some reason....

Did you see those?

[meejah]

Okay, so this whole "should /foo/bar/ actually be the very same thing as /foo/bar opens up a can of worms (see the suggestion/review comment by @exarkun about the return self in getChild and that maybe it should be a re-direct). There are at least a bunch of tests that treat "/uri/foo/" as if it should be the same thing as "/uri/foo" -- but also they have some query args. twisted.web.util.redirectTo doesn't do query args .. and so before I went and wrote a thing that "does a redirect, but also respects query args" I thought "maybe we SHOULD just delete that?".

So the question is: should /uri/foo/bar/ just be an error / empty? Or should it re-direct to /uri/foo/bar?

edit: the plot thickens, tahoe backup seems to "always" but a trailing-slash on URIs...

[meejah]

I tried to make an alternative to the last commit that does a 307 redirect (instead of making it an error) on urls like /foo/bar/ ... buuut I think twisted's web client has a bug (that it doesn't do POST redirects ever) and also at least one thing in Tahoe is using httplib (!!!) as a client which also won't re-direct.

So, I could still make it a re-direct instead of an error fairly easily, but a lot of the other changes to tahoe backup and the tests (in that last commit) would have to stay. @exarkun opinions?

[exarkun]

Thanks. This was a slog to read. I'm sure it was worse to write. ;) Overall my impression is that we have a ton of work left to do to clean up Tahoe-LAFS' web interface... However this appears to be a significant and valid step towards eliminating the Nevow dependency ...

So, please have a look at the very minor inline comments and then .. I want to say "feel free to merge" but I guess we have to do something about CI.

None of the configured GitHub Actions have run for this PR. Is it older than them? I don't think so. I read something about how both repos (original and fork) have to have GitHub Actions turned on for them to run ... but the thread was fairly unclear so I'm not sure that's it. The simplest thing might just be to prefer in-repo PRs instead of PRs from forks... I dunno. After reading all that web code I can't also think about minimizing GitHub workflow weirdness.