New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDN mime type analysis #244

Closed
metadave opened this Issue May 11, 2017 · 8 comments

Comments

Projects
3 participants
@metadave
Member

metadave commented May 11, 2017

The unique list of mime types in SCL3 httpd config are as follows:

AddType application/json .json
AddType application/octet-stream .dump
AddType application/ogg .ogx
AddType application/x-compress .Z
AddType application/x-data .data
AddType application/x-gzip .gz .tgz
AddType application/x-pkcs7-crl    .crl
AddType application/x-x509-ca-cert .crt
AddType application/x-xpinstall .xpi
AddType audio/ogg .ogg
AddType audio/ogg .ogg .spx
AddType image/png .png
AddType image/svg+xml .svg
AddType image/svg+xml .svg .svgz
AddType image/x-icon .ico
AddType text/cache-manifest .appcache
AddType text/html .shtml
AddType video/ogg .ogv
AddType video/webm .webm
AddType x-java-archive .jar

Django allowed_types are listed here

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock May 13, 2017

Contributor

Two places where this might need to be migrated:

When Whitenoise serves a file (from static), it uses a built-in list to determine MIME types. There's also a WHITENOISE_MIMETYPES setting, but it's unclear to me if this replaces the list or augments it without reading the Whitenoise code, and we're currently not using it. It is unclear to me if the Apache list affects files served by Whitenoise. I'm 80% sure that it does not. A few headless integration tests would confirm this.

The Apache mime types definitely come into play when serving user uploads and legacy media files. Those appear to be covered in task #241, where the aliases for file serving are enumerated. I doubt that this list is well-synced with the files served by MDN. There are probably file extensions missing from this list (does that then use system mime types?), and there are probably file extensions here not used by MDN (maybe copied from Bugzilla).

Contributor

jwhitlock commented May 13, 2017

Two places where this might need to be migrated:

When Whitenoise serves a file (from static), it uses a built-in list to determine MIME types. There's also a WHITENOISE_MIMETYPES setting, but it's unclear to me if this replaces the list or augments it without reading the Whitenoise code, and we're currently not using it. It is unclear to me if the Apache list affects files served by Whitenoise. I'm 80% sure that it does not. A few headless integration tests would confirm this.

The Apache mime types definitely come into play when serving user uploads and legacy media files. Those appear to be covered in task #241, where the aliases for file serving are enumerated. I doubt that this list is well-synced with the files served by MDN. There are probably file extensions missing from this list (does that then use system mime types?), and there are probably file extensions here not used by MDN (maybe copied from Bugzilla).

@metadave metadave self-assigned this May 15, 2017

@metadave metadave moved this from Queued to MDN Queued in [Deprecated] MozMEAO SRE Jun 20, 2017

@metadave metadave modified the milestones: MDN AWS Architecture Eval & Recommendation, MDN implementation Jun 22, 2017

@metadave metadave removed this from MDN Queued in [Deprecated] MozMEAO SRE Jul 26, 2017

@metadave metadave added this to Backlog in MDN AWS Migration Jul 26, 2017

@metadave metadave modified the milestone: MDN implementation Aug 8, 2017

@metadave metadave moved this from Backlog to Queued (Max 6) in MDN AWS Migration Aug 11, 2017

@metadave metadave removed their assignment Sep 5, 2017

@metadave

This comment has been minimized.

Show comment
Hide comment
@metadave

metadave Sep 6, 2017

Member

@jwhitlock @escattone I'm unsure if recent PR's have addressed these mime type issues.

Member

metadave commented Sep 6, 2017

@jwhitlock @escattone I'm unsure if recent PR's have addressed these mime type issues.

@jwhitlock jwhitlock self-assigned this Sep 14, 2017

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock Sep 14, 2017

Contributor

I'll do some testing with curl on https://mdn-mm.moz.works. Some ideas for where to look:

  • attachments
  • contribute.json
  • diagrams
  • presentations
  • samples
Contributor

jwhitlock commented Sep 14, 2017

I'll do some testing with curl on https://mdn-mm.moz.works. Some ideas for where to look:

  • attachments
  • contribute.json
  • diagrams
  • presentations
  • samples
@metadave

This comment has been minimized.

Show comment
Hide comment
@metadave

metadave Sep 14, 2017

Member

much appreciated @jwhitlock!

Member

metadave commented Sep 14, 2017

much appreciated @jwhitlock!

@jwhitlock jwhitlock moved this from Queued (Max 6) to In Progress (Max 6) in MDN AWS Migration Sep 14, 2017

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock Sep 14, 2017

Contributor

After extensively looking at the files we're serving vs how they are served from SCL3, I think we're in good shape.

I think a lot of the Apache rules were left over from the demo studio, or copied from some other project.

Attachments have the MIME type stored in the database, and set it as the Content-Type: header. I only checked a few before determining that the extension had nothing to do with the MIME type.

I looked for other files:

I couldn't find examples files for these rules, so we'll ignore them.

AddType application/octet-stream .dump
AddType application/ogg .ogx
AddType application/x-compress .Z
AddType application/x-data .data
AddType application/x-gzip .gz .tgz
AddType application/x-pkcs7-crl    .crl
AddType application/x-x509-ca-cert .crt
AddType image/x-icon .ico
AddType text/cache-manifest .appcache
AddType text/html .shtml
AddType video/ogg .ogv
AddType video/webm .webm
AddType x-java-archive .jar
Contributor

jwhitlock commented Sep 14, 2017

After extensively looking at the files we're serving vs how they are served from SCL3, I think we're in good shape.

I think a lot of the Apache rules were left over from the demo studio, or copied from some other project.

Attachments have the MIME type stored in the database, and set it as the Content-Type: header. I only checked a few before determining that the extension had nothing to do with the MIME type.

I looked for other files:

I couldn't find examples files for these rules, so we'll ignore them.

AddType application/octet-stream .dump
AddType application/ogg .ogx
AddType application/x-compress .Z
AddType application/x-data .data
AddType application/x-gzip .gz .tgz
AddType application/x-pkcs7-crl    .crl
AddType application/x-x509-ca-cert .crt
AddType image/x-icon .ico
AddType text/cache-manifest .appcache
AddType text/html .shtml
AddType video/ogg .ogv
AddType video/webm .webm
AddType x-java-archive .jar

@jwhitlock jwhitlock closed this Sep 14, 2017

@jwhitlock jwhitlock moved this from In Progress (Max 6) to Review in MDN AWS Migration Sep 14, 2017

@metadave

This comment has been minimized.

Show comment
Hide comment
@metadave

metadave Sep 14, 2017

Member

👏 thank you!

Member

metadave commented Sep 14, 2017

👏 thank you!

@jwhitlock

This comment has been minimized.

Show comment
Hide comment
@jwhitlock

jwhitlock Sep 14, 2017

Contributor

Script at https://gist.github.com/jwhitlock/ba528e8f6748992734bc31e251fae3e2, in case I need to do similar work again

Contributor

jwhitlock commented Sep 14, 2017

Script at https://gist.github.com/jwhitlock/ba528e8f6748992734bc31e251fae3e2, in case I need to do similar work again

@escattone

This comment has been minimized.

Show comment
Hide comment
@escattone

escattone Sep 14, 2017

Member

Nice work @jwhitlock, thanks!

Member

escattone commented Sep 14, 2017

Nice work @jwhitlock, thanks!

@metadave metadave moved this from Review to Complete in MDN AWS Migration Sep 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment