Remove code replaced by El Proxito and stateless servers #6535

humitos · 2020-01-18T11:20:14Z

After working on #6326 and realizing that PR will introduce a breaking change, I started playing around by removing more code that we know is not going to be needed once El Proxito is fully rolled out. The idea is to make just one breaking change with all of these remotions together if that is possible. This PR removes these,

Syncers: all the files generated by the builders are copied directly from there to the Django Storage backend configured
Some web server tasks: there is no need to remove, sync or copy files anymore since these servers won't have any documentation or state on them
Symlink code: files are not served from disk anymore. They are served from storage backend via El Proxito
Subdomain and SingleVersion logic + urls: this is now handled by El Proxito

This PR will introduce a big breaking change and force core team and contributors to use our docker-compose based solution for their development environment.

* `remove_dirs` is not needed to be executed anymore on webs because all the artifacts are saved on django storage backend now. So, this task is being replaced by `clean_project_resources`. * `move_files` is not needed because we don't need to sync files anymore to the web servers. Everything is uploaded from build servers to the django storage backend now. * broadcast finalization steps are replaced by only a call to `filefy`. The rest are not required anymore. * `update_static_metadata` won't be needed because the redirection done by the Flask app it will be replaced by El Proxito. * Syncers are removed completely. This copying is happening on build servers now and pushing the files to django storage backend.

We are serving documentation from El Proxito now and these middlewares are not required. The ability to server at `/docs/` is removed as well.

ericholscher · 2020-01-21T16:33:31Z

Some web server tasks: there is no need to remove, sync or copy files anymore since these servers won't have any documentation or state on them

We still need to run these tasks, no? Just not on the web servers but in cloud storage?

humitos · 2020-01-21T20:50:39Z

We still need to run these tasks, no? Just not on the web servers but in cloud storage?

This is done inside the UpdateDocsTaskStep that runs in the builder now, at this method

readthedocs.org/readthedocs/projects/tasks.py

Lines 784 to 796 in 0bdfeb6

    
               def store_build_artifacts( 
        
                       self, 
        
                       html=False, 
        
                       localmedia=False, 
        
                       search=False, 
        
                       pdf=False, 
        
                       epub=False, 
        
               ): 
        
                   """ 
        
                   Save build artifacts to "storage" using Django's storage API. 
        
                   The storage could be local filesystem storage OR cloud blob storage 
        
                   such as S3, Azure storage or Google Cloud Storage.

(there is a call to storage.sync_directory inside that method)

stale · 2020-03-06T20:56:53Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

humitos · 2020-03-13T21:43:19Z

Let's un-stale this PR for now. We are getting close to be able to remove all this code. We will be pinged back by stale bot in a month or so.

stale · 2020-04-28T03:22:27Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

…humitos/remove-web-related-tasks

humitos · 2020-04-28T11:06:55Z

I updated the PR with the latest changes. We are getting closer to be able to remove all this code.

ericholscher · 2020-04-28T14:49:28Z

Yea, I'd like to get the symlink code removed this week if possible. It should make our tests a lot less spammy.

ericholscher · 2020-04-29T16:54:10Z

@humitos looks like CI is unhappy with this, so still some work to do.

ericholscher

Excited for this :)

ericholscher · 2020-04-29T17:10:22Z

readthedocs/projects/views/private.py

-    def form_valid(self, form):
-        broadcast(
-            type='app',
-            task=tasks.symlink_subproject,


I think a lot of the places we were calling symlink code we actually want to bust the cache for the project as well. Something to consider, though maybe we want to add that logic in a new PR after this.

I agree. In this particular case it was called when de-activating a version. The following line (which I didn't add) triggers a clean_project_resources which removes the version from the storage. Seems like someone already thought about this 😄

It seems I already added some ones where I found the same case: https://github.com/readthedocs/readthedocs.org/pull/6535/files#diff-6e7074d6dee951393c702a7071bc0f1eR232

ericholscher · 2020-04-29T17:10:54Z

readthedocs/settings/base.py

@@ -190,8 +185,6 @@ def USE_PROMOS(self):  # noqa
        'django.contrib.auth.middleware.AuthenticationMiddleware',
        'django.contrib.messages.middleware.MessageMiddleware',
        'dj_pagination.middleware.PaginationMiddleware',
-        'readthedocs.core.middleware.SubdomainMiddleware',
-        'readthedocs.core.middleware.SingleVersionMiddleware',


I think we replace this in the proxito code, so probably need to update that as well.

ericholscher · 2020-04-29T17:11:15Z

readthedocs/settings/base.py

@@ -577,7 +561,6 @@ def USE_PROMOS(self):  # noqa
    GRAVATAR_DEFAULT_IMAGE = 'https://assets.readthedocs.org/static/images/silhouette.png'  # NOQA
    OAUTH_AVATAR_USER_DEFAULT_URL = GRAVATAR_DEFAULT_IMAGE
    OAUTH_AVATAR_ORG_DEFAULT_URL = GRAVATAR_DEFAULT_IMAGE
-    RESTRICTEDSESSIONS_AUTHED_ONLY = True


This seems unrelated?

Hrm... I'm not sure how this ended up here. I could re-add it, but we don't use it anymore, though. It's from https://github.com/mxsasha/django-restricted-sessions

ericholscher · 2020-04-29T17:11:28Z

readthedocs/settings/proxito/base.py

-            'readthedocs.core.middleware.SubdomainMiddleware'
-        )
-        classes[index] = 'readthedocs.proxito.middleware.ProxitoMiddleware'
+        classes.append('readthedocs.proxito.middleware.ProxitoMiddleware')


…humitos/remove-web-related-tasks

`ln -nsf` is not called anymore when building docs.

We do not use `/docs/` anymore. We have more similar tests under rtd_tests/tests/test_resolver.py that are better.

Most of these files were migrated to `readthedocs/proxito/tests/test_old_redirects.py` and they are using the new settings and behaves according production. We could remove this file (`readthedocs/rtd_tests/tests/test_redirects.py`) completely probably. For now, I'm just removing the tests that are failing, tho

…humitos/remove-web-related-tasks

humitos · 2020-04-29T21:08:49Z

I worked on this to fix all the tests. I found and fixed these:

some popen count changed because we are not running ln -ns anymore
removed resolver tests using /docs/ prefix --they were already migrated to rtd_tests/tests/test_resolver.py already
removed some redirects started failing because they use old /docs/ prefix --they were already migrated to readthedocs/rtd_tests/tests/test_redirects.py already

There are only 2 tests that are failing:

is happening on hg pull, and I'm not sure why yet
private doc serving under /docs/django-kong/ --which I'm not sure if I should remove it directly

ericholscher · 2020-04-29T21:19:14Z

private doc serving under /docs/django-kong/ --which I'm not sure if I should remove it directly

Yea, we shouldn't support those URL's anymore, so we don't need tests for them 👍 If we can migrate the test to the subdomain URL thats good, if there already is one, delete the /docs/ one.

This was only working on /docs/ paths. Now, with El Proxito we are not respecting the privacy level in community site.

ericholscher

I'm 👍 on this. We should merge it soon so that it doesn't diverge much from the main codeline. @readthedocs/core, if folks could take a look today, I think it makes sense to try and get this shipped for next week.

The other option I see if to merge it right after deploy next week -- that way we have some time for it to settle in.

davidfischer

I'm in favor of merging this although I don't think I'm fully qualified to review every aspect. I'm glad to see all the syncer code go, however.

davidfischer · 2020-05-01T18:29:49Z

readthedocs/urls.py

@@ -22,7 +21,6 @@

 admin.autodiscover()

-handler404 = server_error_404


I think you still want this. This is to handle dashboard 404s too, I believe.

Regarding the 404, we are using the Django default on dashboard: https://docs.djangoproject.com/en/2.2/ref/views/#django.views.defaults.page_not_found --It keeps showing our custom 404.html Maze Found.

Our custom handler is not more required because we are not doing any redirect now, just showing the Maze.

I just checked that running the code with DEBUG=False still shows the Maze Found page on any 404 page under the dashboard.

davidfischer · 2020-05-01T18:31:04Z

readthedocs/urls.py

@@ -46,8 +44,6 @@
    url(r'^accounts/gold/', include('readthedocs.gold.urls')),
    # For redirects
    url(r'^builds/', include('readthedocs.builds.urls')),
-    # For testing the 404's with DEBUG on.
-    url(r'^404/$', handler404),


These enable the following URLs:

https://readthedocs.org/404/

https://readthedocs.org/500/

These are pretty useful in debugging and maybe they should be in the debug_urls. I don't think we should remove just one of them though.

I didn't remove 500. That is still working.

/404/ will use the default Django handler and it should keep working normally as any other not found URL.

Showing it when DEBUG=False fails because we are not registering the URLs on DEBUG=False. Also, leaving just the default value does not work because it checks for the IP to be an INTERNAL_IP, which is not the case on Docker.

…humitos/remove-web-related-tasks

stsewd · 2020-05-04T18:00:30Z

This can be removed too

readthedocs.org/readthedocs/projects/models.py

Line 582 in 0b4eadd

def get_production_media_path(self, type_, version_slug, include_file=True):

stsewd

Coverage actually decreased to 81.2% ha :(

https://codecov.io/gh/readthedocs/readthedocs.org/pull/6535/diff

humitos · 2020-05-06T11:22:49Z

I'm going to merge this so we have 1 week testing it before deploying. Yay! 🎉

humitos added 3 commits January 18, 2020 12:03

Remove symlink code related

34683dc

Remove Subdomain and SingleVersion middlewares

9518912

We are serving documentation from El Proxito now and these middlewares are not required. The ability to server at `/docs/` is removed as well.

humitos added the PR: work in progress Pull request is not ready for full review label Jan 18, 2020

Remove more settings, command and test class

4d54072

humitos force-pushed the humitos/remove-web-related-tasks branch from dc5448b to 4d54072 Compare January 18, 2020 11:54

This was referenced Jan 25, 2020

Refactor tasks files #3775

Closed

Cleanup how path are handled (dev/production) #4119

Closed

Remove webs from broadcast list #6592

Closed

stale bot added the Status: stale Issue will be considered inactive soon label Mar 6, 2020

humitos mentioned this pull request Mar 11, 2020

Remove fake broadcast operations against builder queues #6775

Closed

stale bot closed this Mar 13, 2020

humitos reopened this Mar 13, 2020

stale bot removed the Status: stale Issue will be considered inactive soon label Mar 13, 2020

humitos mentioned this pull request Mar 14, 2020

Only copy HTML build output to web servers #6326

Closed

stale bot added the Status: stale Issue will be considered inactive soon label Apr 28, 2020

Merge branch 'master' of github.com:readthedocs/readthedocs.org into …

53941f4

…humitos/remove-web-related-tasks

stale bot removed the Status: stale Issue will be considered inactive soon label Apr 28, 2020

humitos removed the PR: work in progress Pull request is not ready for full review label Apr 28, 2020

ericholscher reviewed Apr 29, 2020

View reviewed changes

humitos added 2 commits April 29, 2020 19:43

Merge branch 'master' of github.com:readthedocs/readthedocs.org into …

5d31d99

…humitos/remove-web-related-tasks

Remove symlinks from toctree

032a4b2

humitos added 4 commits April 29, 2020 20:54

Reduce call count in 2

7ab30d7

`ln -nsf` is not called anymore when building docs.

Remove merge conflict

05b7063

Remove old test for resolver

cdb0849

We do not use `/docs/` anymore. We have more similar tests under rtd_tests/tests/test_resolver.py that are better.

humitos marked this pull request as ready for review April 29, 2020 20:27

humitos added 2 commits April 29, 2020 22:28

Merge branch 'master' of github.com:readthedocs/readthedocs.org into …

935dd6c

…humitos/remove-web-related-tasks

Delete project test fixed

fc4459d

humitos added 3 commits April 30, 2020 10:26

Lint

9c722d8

We do not support private versions anymore in .org

d515901

This was only working on /docs/ paths. Now, with El Proxito we are not respecting the privacy level in community site.

Fix mercurial test

eeaf144

ericholscher approved these changes May 1, 2020

View reviewed changes

davidfischer reviewed May 1, 2020

View reviewed changes

humitos added 2 commits May 4, 2020 14:26

Show debug toolbar only on DEBUG

317428d

Showing it when DEBUG=False fails because we are not registering the URLs on DEBUG=False. Also, leaving just the default value does not work because it checks for the IP to be an INTERNAL_IP, which is not the case on Docker.

Merge branch 'master' of github.com:readthedocs/readthedocs.org into …

4338b99

…humitos/remove-web-related-tasks

stsewd mentioned this pull request May 4, 2020

Remove usage of project.privacy_level #7013

Merged

Remove unneeded methods to get local paths

26c9393

stsewd approved these changes May 4, 2020

View reviewed changes

humitos merged commit 238ef8b into master May 6, 2020

humitos deleted the humitos/remove-web-related-tasks branch May 6, 2020 11:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove code replaced by El Proxito and stateless servers #6535

Remove code replaced by El Proxito and stateless servers #6535

humitos commented Jan 18, 2020 •

edited

ericholscher commented Jan 21, 2020

humitos commented Jan 21, 2020

stale bot commented Mar 6, 2020

humitos commented Mar 13, 2020

stale bot commented Apr 28, 2020

humitos commented Apr 28, 2020

ericholscher commented Apr 28, 2020

ericholscher commented Apr 29, 2020

ericholscher left a comment

ericholscher Apr 29, 2020

humitos Apr 29, 2020

humitos Apr 29, 2020

ericholscher Apr 29, 2020

ericholscher Apr 29, 2020

humitos Apr 29, 2020

ericholscher Apr 29, 2020

humitos commented Apr 29, 2020 •

edited

ericholscher commented Apr 29, 2020 •

edited

ericholscher left a comment •

edited

davidfischer left a comment

davidfischer May 1, 2020

humitos May 4, 2020

humitos May 4, 2020

davidfischer May 1, 2020

humitos May 4, 2020

humitos May 4, 2020

stsewd commented May 4, 2020

stsewd left a comment

humitos commented May 6, 2020

		@@ -22,7 +21,6 @@

		admin.autodiscover()

		handler404 = server_error_404

Remove code replaced by El Proxito and stateless servers #6535

Remove code replaced by El Proxito and stateless servers #6535

Conversation

humitos commented Jan 18, 2020 • edited

ericholscher commented Jan 21, 2020

humitos commented Jan 21, 2020

stale bot commented Mar 6, 2020

humitos commented Mar 13, 2020

stale bot commented Apr 28, 2020

humitos commented Apr 28, 2020

ericholscher commented Apr 28, 2020

ericholscher commented Apr 29, 2020

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Apr 29, 2020 • edited

ericholscher commented Apr 29, 2020 • edited

ericholscher left a comment • edited

Choose a reason for hiding this comment

davidfischer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented May 4, 2020

stsewd left a comment

Choose a reason for hiding this comment

humitos commented May 6, 2020

humitos commented Jan 18, 2020 •

edited

humitos commented Apr 29, 2020 •

edited

ericholscher commented Apr 29, 2020 •

edited

ericholscher left a comment •

edited