New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate MDN Apache Alias rules #241

Closed
metadave opened this Issue May 11, 2017 · 7 comments

Comments

Projects
3 participants
@metadave
Member

metadave commented May 11, 2017

The following rewrites and aliases will have to be accounted for in Kuma:

# Links to FTP'ed code samples and examples
RewriteRule ^patches(.*) data/www/patches$1 [L]
RewriteRule ^presentations(.*) data/www/presentations$1 [L]
RewriteRule ^samples(.*) data/www/samples$1 [L]
RewriteRule ^diagrams(.*) data/www/diagrams$1 [L]
RewriteRule ^web-tech(.*) data/www/web-tech$1 [L]
RewriteRule ^css(.*) data/www/css$1 [L]

# Rewrites to robots & sitemaps
RewriteRule ^robots.txt$ media/robots.txt [L]
RewriteRule ^sitemap.xml$ media/sitemap.xml [L]
RewriteRule ^sitemaps/([\w\-]*)/sitemap.xml$ media/sitemaps/$1/sitemap.xml [L]

mozilla/domains/developer.cdn.mozilla.net.conf
28:    Alias /media /data/www/developer.mozilla.org/kuma/media
29:    Alias /admin-media /data/www/developer.mozilla.org/kuma/vendor/src/django/django/contrib/admin/media

conf/httpd.conf
561:Alias /icons/ "/var/www/icons/"
873:Alias /error/ "/var/www/error/"

conf/httpd.conf.rpmnew
551:Alias /icons/ "/var/www/icons/"
576:ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
855:Alias /error/ "/var/www/error/"

mozilla/domains/developer.mozilla.org.conf
72:    Alias /media /data/www/developer.mozilla.org/kuma/media
73:    Alias /admin-media /data/www/developer.mozilla.org/kuma/vendor/src/django/django/contrib/admin/media
75:    Alias /presentations /data/www/presentations
76:    Alias /samples /data/www/samples
77:    Alias /diagrams /data/www/diagrams
90:    WSGIScriptAlias /mwsgi /data/www/developer.mozilla.org/kuma/wsgi/kuma.wsgi

parent issue

@metadave

This comment has been minimized.

Show comment
Hide comment
@metadave

metadave May 11, 2017

Member

@jwhitlock @jgmize @escattone Are these best handled by whitenoise?

Member

metadave commented May 11, 2017

@jwhitlock @jgmize @escattone Are these best handled by whitenoise?

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock May 13, 2017

Contributor

😢

Many of these are Apache-only and can be ignored. Some could be served by Whitenoise, but shouldn't, because it will slow down engine startup while Whitenoise calculates hashes for very large, seldom-accessed files. Others are dynamically created and Whitenoise says it is a bad idea to serve them.

I'm not sure what the best option is. Maybe redirect all these file-serving URLs to https://mdn.mozillademos.org/ or developer.cdn.mozilla.net, and run nginx there?

Details:

  • RewriteRule ^patches(.*) data/www/patches$1 [L] - No such files in production, can ignore
  • RewriteRule ^presentations(.*) data/www/presentations$1 [L] - PDFs, videos, and slides from 2005 - 2009, no new ones expected. I eventually want to redirect to S3 or similar, Django / Whitenoise is OK for now.
  • RewriteRule ^samples(.*) data/www/samples$1 [L] - Demonstration code from 2005 - 2012, no new ones expected. There was a project around 2015 to redirect these to current MDN pages and examples. Django / Whitenoise is OK for now.
  • RewriteRule ^diagrams(.*) data/www/diagrams$1 [L] - What was someone's idea of the Mozilla workflow in 2010? Now you know. Django / Whitenoise is OK.
  • RewriteRule ^web-tech(.*) data/www/web-tech$1 [L] - No such files in production, can ignore
  • RewriteRule ^css(.*) data/www/css$1 [L] - No such files in production, can ignore
  • RewriteRule ^robots.txt$ media/robots.txt [L] - This needs to be re-written as a Django view, because it is different in production and staging.
  • RewriteRule ^sitemap.xml$ media/sitemap.xml [L] - Generated by a periodic task. Whitenoise is not appropriate for generated files, may have to be a Django view or other.
  • RewriteRule ^sitemaps/([\w\-]*)/sitemap.xml$ media/sitemaps/$1/sitemap.xml [L] - Generated by a periodic task, not OK for Whitenoise
  • Alias /media /data/www/developer.mozilla.org/kuma/media - Base folder for user uploads and generated files, not appropriate for Whitenoise.
  • Alias /admin-media /data/www/developer.mozilla.org/kuma/vendor/src/django/django/contrib/admin/media - Django 1.3 and earlier used this, not applicable in Django 1.4 and later (we're on 1.8).
  • Alias /icons/ "/var/www/icons/" - Apache icons, not needed by Kuma
  • Alias /error/ "/var/www/error/" - Apache error pages, not needed by Kuma
  • ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" - Apache boilerplate, not needed by Kuma
  • WSGIScriptAlias /mwsgi /data/www/developer.mozilla.org/kuma/wsgi/kuma.wsgi - Apache speak for "/mwsgi is my name for Kuma", not a file served by Django / Whitenoise.
Contributor

jwhitlock commented May 13, 2017

😢

Many of these are Apache-only and can be ignored. Some could be served by Whitenoise, but shouldn't, because it will slow down engine startup while Whitenoise calculates hashes for very large, seldom-accessed files. Others are dynamically created and Whitenoise says it is a bad idea to serve them.

I'm not sure what the best option is. Maybe redirect all these file-serving URLs to https://mdn.mozillademos.org/ or developer.cdn.mozilla.net, and run nginx there?

Details:

  • RewriteRule ^patches(.*) data/www/patches$1 [L] - No such files in production, can ignore
  • RewriteRule ^presentations(.*) data/www/presentations$1 [L] - PDFs, videos, and slides from 2005 - 2009, no new ones expected. I eventually want to redirect to S3 or similar, Django / Whitenoise is OK for now.
  • RewriteRule ^samples(.*) data/www/samples$1 [L] - Demonstration code from 2005 - 2012, no new ones expected. There was a project around 2015 to redirect these to current MDN pages and examples. Django / Whitenoise is OK for now.
  • RewriteRule ^diagrams(.*) data/www/diagrams$1 [L] - What was someone's idea of the Mozilla workflow in 2010? Now you know. Django / Whitenoise is OK.
  • RewriteRule ^web-tech(.*) data/www/web-tech$1 [L] - No such files in production, can ignore
  • RewriteRule ^css(.*) data/www/css$1 [L] - No such files in production, can ignore
  • RewriteRule ^robots.txt$ media/robots.txt [L] - This needs to be re-written as a Django view, because it is different in production and staging.
  • RewriteRule ^sitemap.xml$ media/sitemap.xml [L] - Generated by a periodic task. Whitenoise is not appropriate for generated files, may have to be a Django view or other.
  • RewriteRule ^sitemaps/([\w\-]*)/sitemap.xml$ media/sitemaps/$1/sitemap.xml [L] - Generated by a periodic task, not OK for Whitenoise
  • Alias /media /data/www/developer.mozilla.org/kuma/media - Base folder for user uploads and generated files, not appropriate for Whitenoise.
  • Alias /admin-media /data/www/developer.mozilla.org/kuma/vendor/src/django/django/contrib/admin/media - Django 1.3 and earlier used this, not applicable in Django 1.4 and later (we're on 1.8).
  • Alias /icons/ "/var/www/icons/" - Apache icons, not needed by Kuma
  • Alias /error/ "/var/www/error/" - Apache error pages, not needed by Kuma
  • ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" - Apache boilerplate, not needed by Kuma
  • WSGIScriptAlias /mwsgi /data/www/developer.mozilla.org/kuma/wsgi/kuma.wsgi - Apache speak for "/mwsgi is my name for Kuma", not a file served by Django / Whitenoise.
@metadave

This comment has been minimized.

Show comment
Hide comment
@metadave
Member

metadave commented May 15, 2017

any thoughts @jgmize @escattone @pmac?

@pmac

This comment has been minimized.

Show comment
Hide comment
@pmac

pmac May 15, 2017

Member

it will slow down engine startup while Whitenoise calculates hashes for very large, seldom-accessed files.

I didn't realize that whitenoise did this. I know you can have Django calculate hashes and include said hashes in the filenames of files, but it doesn't recalculate ever, it just stores the map of original file to hashed file in a JSON manifest, and all of that happens at build time. I could be mistaken though depending on your configuration.

RewriteRule ^sitemap.xml$ media/sitemap.xml [L] - Generated by a periodic task. Whitenoise is not appropriate for generated files, may have to be a Django view or other.

Whitenoise is not appropriate for user generated files. As long as the file exists at app startup it should work fine. Whitenoise just won't pick up new files added to the document root but should work fine if the file is subsequently updated (I think).

Member

pmac commented May 15, 2017

it will slow down engine startup while Whitenoise calculates hashes for very large, seldom-accessed files.

I didn't realize that whitenoise did this. I know you can have Django calculate hashes and include said hashes in the filenames of files, but it doesn't recalculate ever, it just stores the map of original file to hashed file in a JSON manifest, and all of that happens at build time. I could be mistaken though depending on your configuration.

RewriteRule ^sitemap.xml$ media/sitemap.xml [L] - Generated by a periodic task. Whitenoise is not appropriate for generated files, may have to be a Django view or other.

Whitenoise is not appropriate for user generated files. As long as the file exists at app startup it should work fine. Whitenoise just won't pick up new files added to the document root but should work fine if the file is subsequently updated (I think).

@metadave metadave self-assigned this May 15, 2017

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock May 15, 2017

Contributor

@pmac is probably right, whitenoise probably doesn't calculate hashes but leaves this up to Django. I have little experience with whitenoise.

Contributor

jwhitlock commented May 15, 2017

@pmac is probably right, whitenoise probably doesn't calculate hashes but leaves this up to Django. I have little experience with whitenoise.

@metadave metadave moved this from Queued to MDN Queued in [Deprecated] MozMEAO SRE Jun 20, 2017

@metadave metadave modified the milestones: MDN AWS Architecture Eval & Recommendation, MDN implementation Jun 22, 2017

@metadave metadave referenced this issue May 15, 2017

Closed

MDN migration #253

@metadave metadave removed this from MDN Queued in [Deprecated] MozMEAO SRE Jul 26, 2017

@metadave metadave added this to Backlog in MDN AWS Migration Jul 26, 2017

@metadave metadave modified the milestone: MDN implementation Aug 8, 2017

@metadave metadave moved this from Backlog to Queued (Max 6) in MDN AWS Migration Aug 11, 2017

@metadave metadave removed their assignment Sep 5, 2017

@metadave

This comment has been minimized.

Show comment
Hide comment
@metadave

metadave Sep 6, 2017

Member

@jwhitlock @escattone I'm unsure if recent PR's have addressed these aliases.

Member

metadave commented Sep 6, 2017

@jwhitlock @escattone I'm unsure if recent PR's have addressed these aliases.

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock Sep 14, 2017

Contributor

Yes, these have been addressed! 😮

Contributor

jwhitlock commented Sep 14, 2017

Yes, these have been addressed! 😮

@jwhitlock jwhitlock closed this Sep 14, 2017

@jwhitlock jwhitlock moved this from Queued (Max 6) to Review in MDN AWS Migration Sep 14, 2017

@metadave metadave moved this from Review to Complete in MDN AWS Migration Sep 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment