Redirects (design doc): improving existing functionality #10825

stsewd · 2023-10-17T00:44:11Z

Didn't include anything about built-in redirects, should I touch that too in this document? Maybe in another one?

📚 Documentation previews 📚

User's documentation (docs): https://docs--10825.org.readthedocs.build/en/10825/

Developer's documentation (dev): https://dev--10825.org.readthedocs.build/en/10825/

stsewd · 2023-10-17T00:44:47Z

Marked this PR as draft, but it's ready to have some feedback. Preview at https://dev--10825.org.readthedocs.build/en/10825/design/redirects.html

ericholscher · 2023-10-20T15:16:30Z

@stsewd I think this is definitely more focused on user-defined redirects. I don't think we have too much to change about our automatic ones, except maybe some changes to domain canonicalization at some point, so we can ignore that for this PR 👍

ericholscher

This is a great document, and I'm 💯 on moving forward with all the various proposed features. I do wonder if we can just remove the entire concept of redirect types, and use wildcards and presets to solve this problem? In particular, the HtmlDir redirects could just be a documented option for an Exact redirect that does *.html -> /. This is also a good use case for prefix wildcards 🤔

docs/dev/design/redirects.rst

ericholscher · 2023-10-20T15:26:18Z

docs/dev/design/redirects.rst

+- Placeholders.
+  I haven't seen users requesting this feature.
+  We can consider adding it in the future.
+  Maybe we can expose the current language and version as placeholders.


I think this is a good idea, and perhaps removes the need for Page redirects, since it's the same as /$lang/$version/$path as an exact redirect?

This will be also useful for custom URLconf in the future when we support changing the position of them, like:

From: /:lang/:version/:path To: /:lang/prefix/:version/:path

I don't think we will be allowing to change the positions of components, we already discussed why this introduces a lot of complexity and doesn't solve user requirements (they only need to add a prefix, which we already suport).

stsewd · 2023-10-23T22:01:15Z

@ericholscher I thought about having just one type of redirect, but the only two types of redirects that we can easily replace are: prefix redirects and HTML URLs to clean URLs redirects.

Why we can't replace page redirects?

We could, but it complicates things, since we will be building on top of exact redirects, we need a way to specify in the from URL that we only want to match the filename.
Having a from URL like */file/name.html will match /en/latest/file/name.html, but it will also match /prefix/file/name.html, and /en/latest/directory/file/name.html, which is unexpected. To fix this, we will have to introduce a few changes:

a) Change the prefix wildcard to match the language, version, and custom prefix (basically leaving the filename only). This works, but then it may be confusing for users, that may expect the wildcard to match everything, this also takes away the use case of redirecting all URLs that end with a given path (*.html -> $wildcard/). But if we don't have another use case for a prefix wildcard, this could work...
b) Use a different type of wildcard for it. For example, $prefix or ~. Then it can be used as /~/file/name.html. Having another type of wildcard adds more confusion IMO.
c) Allow to capture URL components, similar to Clouflare/Netlify. In this case, a user can have a from URL like /$language/$version/$prefix/filename.html (the names can be anything). Implementing this may complicate things, we will probably need to go back to iterate each redirect in order to find a match, and use regex...

Having a page redirect type is easier to explain to users, and they will work just like exact redirects, allowing prefix (if we still have the need for them) and suffix redirects.

Why we can't replace clean URLs to .html URLs redirects?

With the suggestion about normalizing paths that end with slash, it becomes impossible to capture paths that end with /. Having a direct like */ -> $start.html will try to redirect both, foo.html and foo/ to foo.html.html and foo.html. If the file doesn't exist, it will generate an infinite redirect (foo.html.html.html.html...).

ericholscher · 2023-10-24T18:36:01Z

@stsewd Gotcha.. I do think Page redirects are a pretty useful feature of versioning, so it makes sense to have them. Perhaps we could call them Versioned redirects to make it a bit more explicit?

The HTMLDir redirect with the normalization is a bit unfortunate. I could see users wanting to match URLs that end with /, but perhaps not? I think normalizing makes sense to start, unless we have an obvious need for an explicit / match (which I guess the HTMLDir redirect is, but we can special case it..)

stsewd · 2023-10-25T17:10:38Z

There is only one relevant pending decision #10825 (comment)

humitos · 2023-10-26T17:26:47Z

I do wonder if we can just remove the entire concept of redirect types, and use wildcards and presets to solve this problem?

I'd love to remove the "types of redirects" if possible. Mainly because they are hard to explain to users and, once you understand how they work, they are hard to remember. The document starts "Our current implement has 5 types of redirects" which seems scary to me as a user. Why redirects should be so complex? 😅

If redirects scares myself, I'd assume that our users are being even more scared with them 🙃

humitos

I review this document and made some comments. I have no idea how complex the implementation of the things I've said could be but I didn't want to avoid mention them. I'm fine getting a reply like "this is good but it will be too complex to implement/maintain" 👍🏼

In general, apart from the limitations our redirects have, I think it's a feature that currently is hard to explain. I think that's because the "multiple types of redirects" and it seems we could potentially remove the types and only have one that depending on if it contains or not a * / $rest behaves in one or another way.

I will try to take another look at this PR and think deeper on this features and also in the conversation that you are having with Eric here as well and try to provide more feedback.

docs/dev/design/redirects.rst

humitos · 2023-10-26T17:23:07Z

docs/dev/design/redirects.rst

+- Placeholders.
+  I haven't seen users requesting this feature.
+  We can consider adding it in the future.
+  Maybe we can expose the current language and version as placeholders.


This will be also useful for custom URLconf in the future when we support changing the position of them, like:

From: /:lang/:version/:path To: /:lang/prefix/:version/:path

humitos · 2023-10-26T17:25:39Z

docs/dev/design/redirects.rst

+  For example, if we have a redirect with the value ``/foo?blue=1&yellow=2&red=3``,
+  if would be normalized in the DB as ``/foo`` and ``blue=1&red=3&yellow=2``.
+  This implies that the URL to be matched must have the exact same query arguments,
+  it can't have more or less.


This can be solved with a JSONField on the db to save the query arguments. In that case, the order and number of the arguments won't matter.

But matching can't be done with one query even using a JSONField either, matching will need to happen on the Python side.

I'm not sure to understand your reply.

I'm saying that with a JSONField that stores the query arguments you can perform the match of the query arguments at DB level. Once you get those, you can perform the path/URL matching with Python.

This will reduce the complexity of handling all the matching on Python.

If the current URl is ?one&two&three, we don't know which of those arguments we need to match in the rules. A rule can be one, or be two, or one&three.

I think this is solvable by "matching the query arguments from the rule into the DB field" -- no the other way around as you are saying. Example:

.filter(queryargs__haskeys=rule.queryargs.keys())

https://docs.djangoproject.com/en/4.2/ref/contrib/postgres/fields/#has-keys

or we can use contained_by if we want to match specific key, value pairs:

.filter(queryargs__contained_by=rule.queryargs)

https://docs.djangoproject.com/en/4.2/ref/contrib/postgres/fields/#std-fieldlookup-hstorefield.contained_by

oh, didn't know about those queries, I have updated the doc to mention this approach.

stsewd · 2023-10-26T17:57:13Z

I'd love to remove the "types of redirects" if possible. Mainly because they are hard to explain to users and, once you understand how they work, they are hard to remember.

We can reduce them to 3, prefix redirects and .html to clean URLs redirects can be removed (if we allow prefix wildcards). We can't replace page redirects and clean URLs to .html as described in
#10825 (comment).
And trying to replace .html to clean URLs requires us to add support prefix wildcards, which introduces more complexity just for that use case. I'll be fine with just removing prefix redirects for now.

humitos

Thanks for answering all my questions. I think we are ready to move forward with the implementation since the big picture is already discussed. We can define more specific things during the implementation, probably.

Is there anything that's not defined yet that I'm missing?

stsewd · 2023-10-30T17:20:58Z

Is there anything that's not defined yet that I'm missing?

Maybe if we should merge prefix redirects with exact redirects

readthedocs.org/docs/dev/design/redirects.rst

Lines 297 to 298 in 9ebd216

    
           - Merge prefix redirects with exact redirects. 
        
             Prefix redirects are the same as exact redirects with a wildcard at the end.

ericholscher · 2023-10-30T19:06:34Z

Is there anything that's not defined yet that I'm missing?

Maybe if we should merge prefix redirects with exact redirects

readthedocs.org/docs/dev/design/redirects.rst

Lines 297 to 298 in 9ebd216

- Merge prefix redirects with exact redirects.

Prefix redirects are the same as exact redirects with a wildcard at the end.

I'm 👍 on merging them. The fewer types we have the better. I think the HTMLDir-style redirects are not super common, so I'm fine making that functionality a bit less obvious in the UI if it simplifies how we explain it to users.

humitos · 2023-10-31T10:31:51Z

Maybe if we should merge prefix redirects with exact redirects

👍🏼 on merging them.

Redirects (design doc): improving existing functionality

1fdaf55

Ref #4221

stsewd requested review from ericholscher and humitos October 17, 2023 00:44

stsewd added 2 commits October 16, 2023 19:54

Update on page redirects

fd067dc

Local TOC

0412d21

stsewd self-assigned this Oct 19, 2023

ericholscher reviewed Oct 20, 2023

View reviewed changes

Small update

4280536

stsewd added 2 commits October 25, 2023 12:05

Update with decisions

983a92a

Comment about merging all redirect types

d943713

humitos reviewed Oct 26, 2023

View reviewed changes

stsewd added 3 commits October 26, 2023 13:02

GitLab pages + examples

9c73e5d

Normalize paths before saving them

68fa2f9

Use * and :splat syntax

9ebd216

humitos approved these changes Oct 30, 2023

View reviewed changes

stsewd marked this pull request as ready for review October 30, 2023 17:21

stsewd requested a review from a team as a code owner October 30, 2023 17:21

Last updates

1afddab

stsewd merged commit d0d8cba into main Oct 31, 2023
4 of 5 checks passed

stsewd deleted the redirects-design-doc branch October 31, 2023 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redirects (design doc): improving existing functionality #10825

Redirects (design doc): improving existing functionality #10825

stsewd commented Oct 17, 2023 •

edited by github-actions bot

stsewd commented Oct 17, 2023 •

edited

ericholscher commented Oct 20, 2023

ericholscher left a comment •

edited

ericholscher Oct 20, 2023

humitos Oct 26, 2023

stsewd Oct 26, 2023

stsewd commented Oct 23, 2023 •

edited

ericholscher commented Oct 24, 2023

stsewd commented Oct 25, 2023

humitos commented Oct 26, 2023

humitos left a comment

humitos Oct 26, 2023

humitos Oct 26, 2023

stsewd Oct 26, 2023

humitos Oct 30, 2023 •

edited

stsewd Oct 30, 2023

humitos Oct 31, 2023

stsewd Oct 31, 2023

stsewd commented Oct 26, 2023

humitos left a comment

stsewd commented Oct 30, 2023

ericholscher commented Oct 30, 2023

humitos commented Oct 31, 2023

Redirects (design doc): improving existing functionality #10825

Redirects (design doc): improving existing functionality #10825

Conversation

stsewd commented Oct 17, 2023 • edited by github-actions bot

stsewd commented Oct 17, 2023 • edited

ericholscher commented Oct 20, 2023

ericholscher left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Oct 23, 2023 • edited

Why we can't replace page redirects?

Why we can't replace clean URLs to .html URLs redirects?

ericholscher commented Oct 24, 2023

stsewd commented Oct 25, 2023

humitos commented Oct 26, 2023

humitos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos Oct 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Oct 26, 2023

humitos left a comment

Choose a reason for hiding this comment

stsewd commented Oct 30, 2023

ericholscher commented Oct 30, 2023

humitos commented Oct 31, 2023

stsewd commented Oct 17, 2023 •

edited by github-actions bot

stsewd commented Oct 17, 2023 •

edited

ericholscher left a comment •

edited

stsewd commented Oct 23, 2023 •

edited

humitos Oct 30, 2023 •

edited