spec: Add AMP framework hosting guide #27100
Conversation
There was a problem hiding this comment.
This is fantastic! Thanks a lot. Left a few comments / suggestions. @CrystalOnScript can you please review as well.
spec/amp-framework-hosting.md
Outdated
|
|
||
| where | ||
|
|
||
| - `<config>` is one of `canary` or `prod`, indicating whether the build is meant to be served as a canary release or a production release, respectively. When the `--config` flag is omitted, the build system defaults to `prod`. |
There was a problem hiding this comment.
Does it make sense for self-hosting to make a canary build or should devs always do a prod build?
There was a problem hiding this comment.
A big part of the reason I include a description of the config is to explain how an RTV differs from the versions listed at https://github.com/ampproject/amphtml/releases . When I first noticed this discrepancy, I had to scour stack overflow for an answer because I couldn't find it documented anywhere. A hosting guide seems as good a place as any to describe what a runtime version is.
There was a problem hiding this comment.
This puzzles me too. When would I want a canary build? What are the differences between the two?
There was a problem hiding this comment.
@ithinkihaveacat @sebastianbenz Keep in mind that large publishers could take advantage of self-hosting the AMP framework. This feature isn't limited to hobbyists. If a publisher has the means to A/B test prod/canary releases, this guide has enough information to instruct them how the releases should be built or downloaded and distributed.
There was a problem hiding this comment.
Opted for completeness.
spec/amp-framework-hosting.md
Outdated
|
|
||
| The built framework can be found in directory `dist`. The version assigned to the build is in `dist/version.txt` and a listing of all files included in build is in `dist/files.txt`. The framework is ready to be moved to and served from your host. | ||
|
|
||
| ### Option 2: Copy the framework from cdn.ampproject.org |
There was a problem hiding this comment.
Sidenote, we should provide a tool for this. See ampproject/amp-toolbox#642
There was a problem hiding this comment.
A tool would be nice. It would also make sense for the AMP Project to include a bundled zip in the Assets of each release. For example, when you look at the releases of Microsoft Terminal, the source is accompanied by a compiled binary.
There was a problem hiding this comment.
#27726 mentioned in guide for those who might want to watch the issue.
spec/amp-framework-hosting.md
Outdated
| <script async custom-element="amp-carousel" src="https://example.com/amp/v0/amp-carousel-0.1.js"></script> | ||
| ``` | ||
|
|
||
| Versioned URLs are also possible and discussed in section [Serve the framework](#serve-the-framework). |
There was a problem hiding this comment.
Suggestion: add a warning
Important: all scripts must be served from the same origin. It's not possible to mix scripts downloaded from `cdn.ampproject.org` and `example.com`.
spec/amp-framework-hosting.md
Outdated
|
|
||
| The AMP Project has a [weekly release channel](https://amp.dev/documentation/guides-and-tutorials/learn/spec/release-schedule/#weekly), sometime referred to as the "evergreen" release channel, where AMP pages can take advantage of rolling releases without updating the URLs to the AMP framework. In other words, the URLs are "versionless". This is relatively easy to accomplish when hosting the AMP framework yourself. The key is to ensure that the AMP framework hosted from versionless URLs is _also_ available from RTV-specific URLs. For example, if production AMP framework version 200229061636 is available from `https://example.com/amp`, then it must also be available from `https://example.com/amp/rtv/01200229061636`. This suggests an update strategy: first make a new AMP framework version available from RTV-specific URLs and _then_ update the AMP framework available from versionless URLs. | ||
|
|
||
| #### rtv/metadata |
There was a problem hiding this comment.
I don't quite understand the intention of this section.
There was a problem hiding this comment.
When amp-toolbox's optimizer is configured with param ampUrlPrefix and the runtime version is omitted, the /rtv/metadata endpoint is referenced twice: once to determine the version of the runtime at the specified host and once to get the boilerplate CSS for the transformer. So, I think it makes sense to inform users about its existence in case they plan to serve optimized AMP pages.
There was a problem hiding this comment.
Hm. Not sure this is the right place for this. This probably should go into a separate guide. Another option is to put this into a (shorter note):
Note: ...
This will create a callout box on amp.dev.
There was a problem hiding this comment.
Opted to retain section but made it clear that endpoint is optional and listed tools that take advantage of it.
spec/amp-framework-hosting.md
Outdated
|
|
||
| #### Versionless URLs | ||
|
|
||
| The AMP Project has a [weekly release channel](https://amp.dev/documentation/guides-and-tutorials/learn/spec/release-schedule/#weekly), sometime referred to as the "evergreen" release channel, where AMP pages can take advantage of rolling releases without updating the URLs to the AMP framework. In other words, the URLs are "versionless". This is relatively easy to accomplish when hosting the AMP framework yourself. The key is to ensure that the AMP framework hosted from versionless URLs is _also_ available from RTV-specific URLs. For example, if production AMP framework version 200229061636 is available from `https://example.com/amp`, then it must also be available from `https://example.com/amp/rtv/01200229061636`. This suggests an update strategy: first make a new AMP framework version available from RTV-specific URLs and _then_ update the AMP framework available from versionless URLs. |
There was a problem hiding this comment.
Can you also explain which cache expiry headers they should use to serve the version less runtime.
spec/amp-framework-hosting.md
Outdated
|
|
||
| At a minimum, your AMP framework should be hosted in an RTV-specific path. For example, if you expect to host a production AMP framework with version `200229061636` from `https://example.com/amp`, then AMP framework files should be available from `https://example.com/amp/rtv/01200229061636`. As concrete examples, the [AMP runtime](https://amp.dev/documentation/guides-and-tutorials/learn/spec/amphtml/#amp-runtime) should be available at `https://example.com/amp/rtv/01200229061636/v0.js` and component [amp-geo](https://amp.dev/documentation/components/amp-geo/) should be available at `https://example.com/amp/rtv/01200229061636/v0/amp-geo-0.1.js`. | ||
|
|
||
| If hosting a single AMP framework version from an RTV-specific path is your end goal, then you can update the scripts in your AMP pages to download the runtime and components from your host and skip to section [amp-geo hotpatching](#amp-geo-hotpatching). If you expect to update the AMP framework regularly, then updating RTV-specific URLs in AMP pages could be cumbersome and you might be interested in [Versionless URLs](#versionless-urls). |
There was a problem hiding this comment.
Can you add a section explaining that these script should be considered immutable and be served with the corresponding cache headers.
spec/amp-framework-hosting.md
Outdated
| <meta name="amp-geo-api" content="..."> | ||
| ``` | ||
|
|
||
| In each case, the value of the `content` attribute can be an absolute or relative URL, but must be secure (HTTPS). Importantly, **these meta tags must appear before the first `<script>` tag**. For example, if your AMP framework is hosted at `https://example.com/amp`, then your AMPHTML should look similar to: |
There was a problem hiding this comment.
Would be nice if the validator checked for this!
spec/amp-framework-hosting.md
Outdated
|
|
||
| You can host the AMP framework and components from your own server or CDN. This feature has a number of applications. For example, you can... | ||
|
|
||
| - test and demonstrate changes to the framework or components. |
There was a problem hiding this comment.
Does this refer to serving modified versions of the JS? Will that actually be possible? There's a few comments in #25873 saying it should not be possible (aside from amp-geo), but the design document does not refer to this constraint.
There was a problem hiding this comment.
It does. An important bullet point in offering publishers the ability to host the AMP framework is control. This is possible since we've abandoned the idea of mixing a self-hosted AMP framework with certain components from cdn.ampproject.org. FWIW, this is possible today. The only feature under review right now is <meta name="runtime-host"> support. Publishers can opt to build the AMP framework today, specifying a different host in AMP_CONFIG.urls.cdn, and make all the modifications they like. This just documents that feature.
spec/amp-framework-hosting.md
Outdated
|
|
||
| where | ||
|
|
||
| - `<config>` is one of `canary` or `prod`, indicating whether the build is meant to be served as a canary release or a production release, respectively. When the `--config` flag is omitted, the build system defaults to `prod`. |
There was a problem hiding this comment.
This puzzles me too. When would I want a canary build? What are the differences between the two?
- Add another warning about mixing origins - Add cache-control header advice This commit does not address all review items.
There was a problem hiding this comment.
This looks great from my side. @CrystalOnScript can you take a quick look as well.
- Expect absolute URLs in meta and scripts - Version format (13/15-digits) is required - Pages will not yet validate - Shrink some tl;dr sections
|
Guide updated with latest on self-hosting:
Could an OWNER review this document? I believe it is ready for public consumption and can be updated when validation of pages using self-hosted AMP frameworks is figured out. |
spec/amp-framework-hosting.md
Outdated
|
|
||
| - Canary: `00` | ||
| - Production: `01` | ||
| - Experiment: `02`, `03`, `04`, etc. |
There was a problem hiding this comment.
00 - Experimental
01 - Stable
02 - Control
03 - Beta
04 - Nightly
05 - Nightly-Control
/cc @danielrozenberg for opinions on RTV explanation and how much to include here versus https://github.com/ampproject/amphtml/blob/master/contributing/release-schedule.md
There was a problem hiding this comment.
I would be very happy to see this "version vs. rtv" explanation in the release documentation. It's only here because i couldn't find any public-facing documentation (outside of code comments and github issue/pr comments) about what an rtv is.
There was a problem hiding this comment.
Again I'll defer to Daniel here, I believe he's working on some public facing documentation updates to go with Nightly releases as well.
There was a problem hiding this comment.
This information should definitely either be here, or split into its own file. For now I think it's fine to put it here and link to this section if this seems relevant in other files (such as release-schedule.md)
Somewhere in this document (possibly, but not necessarily in this section) you should mention that at the very least to self-serve AMP one must have the 01 files - anything else is only for those interested in performing various A/B tests
Also, adding to @rcebulko's list:
10,11,12- Experiments A, B, and C as defined in https://github.com/ampproject/amphtml/blob/master/build-system/global-configs/experiments-config.json (there's a README.md file in the same directory) with theirenvironmentset toAMP21,23,25- - Experiments A, B, and C with theirenvironmentset toINABOX(ads);20,22, and24can be used as prefixes for control populations (i.e., same traffic volume, but binary equivalent to01)
spec/amp-framework-hosting.md
Outdated
|
|
||
| You can host the AMP framework and components from your own server or CDN. This feature has a number of applications. For example, you can... | ||
|
|
||
| - test and demonstrate changes to the framework or components. |
There was a problem hiding this comment.
supernit: I would put this bullet point last
spec/amp-framework-hosting.md
Outdated
| - deliver the AMP framework in regions where `cdn.ampproject.org` may not be available. | ||
| - serve AMP pages and the framework from the same host, potentially improving content delivery times. | ||
|
|
||
| The AMP Project is looking into options for [validation](https://amp.dev/documentation/guides-and-tutorials/learn/validation-workflow/validate_amp/) of AMP pages that use an AMP framework hosted outside of `cdn.ampproject.org`. As of April 2020, these AMP pages do not pass validation. |
There was a problem hiding this comment.
Is there an issue number that can be linked here?
spec/amp-framework-hosting.md
Outdated
|
|
||
| The AMP Project has adopted the following conventions: | ||
|
|
||
| - Version: the commit time of the last commit in the active branch |
There was a problem hiding this comment.
You can link to #16631 for added context
spec/amp-framework-hosting.md
Outdated
|
|
||
| - Canary: `00` | ||
| - Production: `01` | ||
| - Experiment: `02`, `03`, `04`, etc. |
There was a problem hiding this comment.
This information should definitely either be here, or split into its own file. For now I think it's fine to put it here and link to this section if this seems relevant in other files (such as release-schedule.md)
Somewhere in this document (possibly, but not necessarily in this section) you should mention that at the very least to self-serve AMP one must have the 01 files - anything else is only for those interested in performing various A/B tests
Also, adding to @rcebulko's list:
10,11,12- Experiments A, B, and C as defined in https://github.com/ampproject/amphtml/blob/master/build-system/global-configs/experiments-config.json (there's a README.md file in the same directory) with theirenvironmentset toAMP21,23,25- - Experiments A, B, and C with theirenvironmentset toINABOX(ads);20,22, and24can be used as prefixes for control populations (i.e., same traffic volume, but binary equivalent to01)
spec/amp-framework-hosting.md
Outdated
|
|
||
| #### Versioned URLs | ||
|
|
||
| The simplest option for hosting the AMP framework is to host it from an rtv-specific path. For example, consider AMP framework host `https://example.com/amp`. Production version `200229061636` corresponds to runtime version `01200229061636` and should be hosted from `https://example.com/amp/rtv/01200229061636`. In this case, AMP meta and script URLs would look like the following: |
There was a problem hiding this comment.
Production version → Stable AMP version
spec/amp-framework-hosting.md
Outdated
|
|
||
| #### Versionless URLs | ||
|
|
||
| The AMP Project has a [weekly release channel](https://amp.dev/documentation/guides-and-tutorials/learn/spec/release-schedule/#weekly), sometimes referred to as the "evergreen" release channel. AMP pages utilize this channel by including versionless URLs to AMP scripts and styles. This is relatively easy to accomplish when hosting the AMP framework yourself. The key is to ensure that the AMP framework hosted from versionless URLs is _also_ available from rtv-specific URLs. This suggests an update strategy: first make a new AMP framework version available from rtv-specific URLs and _then_ update the AMP framework available from versionless URLs. For example, if production AMP framework version `200229061636` is available from `https://example.com/amp`, then it must also be available from `https://example.com/amp/rtv/01200229061636`. In this case, AMP meta and script URLs would look like the following: |
spec/amp-framework-hosting.md
Outdated
|
|
||
| - `ampRuntimeVersion` (required) is the current production runtime version of the AMP framework. | ||
| - `ampCssUrl` (optional) is a URL to the boilerplate CSS for the current production runtime version. | ||
| - `canaryPercentage` (optional) indicates the fraction of users who receive the current canary runtime version of the AMP framework instead of the current production runtime version. |
There was a problem hiding this comment.
canary → Experimental (the JSON field name uses the legacy name, but documentation should use the new name)
spec/amp-framework-hosting.md
Outdated
| - `ampRuntimeVersion` (required) is the current production runtime version of the AMP framework. | ||
| - `ampCssUrl` (optional) is a URL to the boilerplate CSS for the current production runtime version. | ||
| - `canaryPercentage` (optional) indicates the fraction of users who receive the current canary runtime version of the AMP framework instead of the current production runtime version. | ||
| - `diversions` (optional) lists active non-production runtime versions (canary and experiments). |
There was a problem hiding this comment.
(canary and experiments) → (e.g., Beta, Experimental, Control. See [#versioning](Versioning) for context)
spec/amp-framework-hosting.md
Outdated
|
|
||
| ### amp-geo hotpatching | ||
|
|
||
| [amp-geo](https://amp.dev/documentation/components/amp-geo/) requires special attention when hosting the AMP framework. When `cdn.ampproject.org` serves any of `amp-geo-0.1.js`, `amp-geo-0.1.mjs`, `amp-geo-latest.js`, or `amp-geo-latest.mjs`, it detects the country where the request originated and replaces string `{{AMP_ISO_COUNTRY_HOTPATCH}}` with the ISO 3166-1 alpha-2 country code (two ascii letter characters) followed by 26 spaces to maintain the string length. Ideally, when hosting the AMP framework, your content distribution platform would perform the same manipulation. |
There was a problem hiding this comment.
Oh, I didn't address this. I still need to work with the amp-geo folks to update the API fallback. I didn't realize they'd committed subdivision support. Notice this also isn't updated yet: https://github.com/ampproject/amphtml/blob/master/spec/amp-cache-guidelines.md#guidelines-adding-a-new-cache-to-the-amp-ecosystem
There was a problem hiding this comment.
Revised with current hotpatch logic.
|
Thank you for the review @danielrozenberg ! I incorporated changes for each of your comments. Let me know if they meet your expectations. |
spec/amp-framework-hosting.md
Outdated
| @@ -66,7 +75,7 @@ The built framework can be found in directory `dist`. The version assigned to th | |||
|
|
|||
| If you have advanced hosting capabilities or would like to manually assign a version, `gulp dist` accepts these flags (among others): | |||
|
|
|||
| - `--config`: Indicate the release type, production (`prod`) or canary (`canary`). Defaults to `prod`. | |||
| - `--config`: Indicate the release type, stable (`prod`) or experimental (`canary`). Defaults to `prod`. | |||
There was a problem hiding this comment.
This is the one case where "production" and "canary" are the correct terms - these refer to which config file is prepended to v0.js (and other base runtimes such as amp4ads-v0.js) and those are referred to as production and canary
spec/amp-framework-hosting.md
Outdated
|
|
||
| #### Update URLs config | ||
|
|
||
| Modify [`src/config.js`](https://github.com/ampproject/amphtml/blob/master/src/config.js): |
There was a problem hiding this comment.
Shouldn't we be requesting they modify the AMP_CONFIG injected when creating an RTV? That way you don't need to maintain the modified config.js with possibly conflicting upstream changes.
There was a problem hiding this comment.
I'm not sure I follow. Are you saying there's a way to define properties in AMP_CONFIG during build (other than by directly editing config.js)? Or are you suggesting AMP pages do something like the following:
<script>AMP_CONFIG = { cdnUrl: 'https://example.com' };</script>
The latter example works well in PWAs, where validation is not a concern (see example self-hosted amp in pwa), but wouldn't work if we hope to support validation of these AMP pages in the future.
There was a problem hiding this comment.
@jridgewell - this is the last comment about which I'm unsure of how to implement your suggestion.
There was a problem hiding this comment.
So the process for generating an RTV is prepending a configuration before the v0.js binary. @rsimha should have the info on where we concat it together.
There was a problem hiding this comment.
Just noticed this thread. The approach you've chosen in #27977 is on the right track if you're looking to inject custom config values at build time.
There was a problem hiding this comment.
Updated, thanks for the suggestion and help.
spec/amp-framework-hosting.md
Outdated
|
|
||
| When you request `amp-geo-0.1.js` from `cdn.ampproject.org` using an HTTP client, the CDN detects the country where the request originated and patches `amp-geo-0.1.js` on-the-fly. This patch needs to be reversed to ensure users are not all assigned the same country when amp-geo loads. | ||
|
|
||
| When `cdn.ampproject.org` serves `amp-geo-0.1.js`, it replaces string `{{AMP_ISO_COUNTRY_HOTPATCH}}` with an ISO 3166-1 country code or an ISO 3166-2 country-subdivision code, followed by enough spaces to maintain the length of the string being replaced. Reversal of this patch can be accomplished by a RegEx replacement: search for `/[a-zA-Z]{2}(?:-[a-zA-Z0-9]{1,3} {22,24}| {26})/` and replace with `{{AMP_ISO_COUNTRY_HOTPATCH}}`. |
There was a problem hiding this comment.
it replaces string
{{AMP_ISO_COUNTRY_HOTPATCH}}with an ISO 3166-1 country code or an ISO 3166-2 country-subdivision code
Technical nits: We don't insert a Country Code if we can't geolocate. If we can, then we insert it. And if we can geolocate, we'll maybe insert a Subdivision, too. So, the format is either:
"{{AMP_ISO_COUNTRY_HOTPATCH}}" // should never happen from CDN
" " // 28 spaces
"CC " // 26 spaces
"CC CC-SD " // 20 spaces
So a regex might be something like / {28}|[A-Z]{2} {26}|[A-Z]{2} [A-Z]{2}-[A-Z]{2} {20}/i
There was a problem hiding this comment.
Thanks for clarifying this. I didn't realize the ISO 3166-2 country-subdivision code is inserted independent of the country code. I just tested in California and I do see
"us us-ca "
I'm under the impression that broader subdivision support will be possible one day, so I'm leaning towards
/ {28}|[a-zA-Z]{2} {26}|[a-zA-Z]{2} [a-zA-Z]{2}-[a-zA-Z0-9]{1,3} {19,21}/
The {1,3} and {19,21} look a little fuzzy, but they greedy-match, so there wouldn't be any whitespace leftover with a code like fj-w. Note: I'm avoiding //i because some folks might not use JavaScript.
There was a problem hiding this comment.
Pretty much all regex engines support the case-insensitive flag, and it really shrinks the regex itself and makes it a bit easier to read, so I think //i should be fine/clear enough.
|
Barring any change requests over the next 24hrs, I’ll go ahead and merge this on May 6. |
|
Thanks a lot for writing this @mdmower ! |
* spec: Add AMP framework hosting guide * Fix URLs * Prettier * Partially address reviews - Add another warning about mixing origins - Add cache-control header advice This commit does not address all review items. * Update version flag, minor reorg * Minor updates * Fix heading * Fix serving framework links and update flags presentation * Rename tool download-runtime --> runtime-download * Update guide with latest on self-hosting - Expect absolute URLs in meta and scripts - Version format (13/15-digits) is required - Pages will not yet validate - Shrink some tl;dr sections * Minor tweaks * Remove redundant info * Minorestest tweak * Lint fixes * Revisions from review * Revisions from review * Subdivision part of ISO 3166-2 is alphanumeric * Subdivision part of ISO 3166-2 has variable length * Revisions from review * Update build instructions * Update for changes in versioning
ampprojectbot
added
PR use: In Beta / Experimental
PR use: In Stable
and removed
PR use: In Stable
labels
Add a guide for hosting the AMP framework
Depends on #26829