Skip to content

Commit

Permalink
Generate package-locks for demo serving (#1220)
Browse files Browse the repository at this point in the history
This change implements a package lock generator. This allows modules in a demo to be loaded using versions based on the resolved package lock result.

The package lock is generated by copying the `package.json` file, placing it an isolated temporary directory and running `npm install` in a separate process. Copying the `package.json` file is required to support both `dependencies` and `devDependencies`, which is a common use case wrt demos. This is an expensive operation which depends on the transitive dependency size. As such, there are number of performance improvements built in:
 * **Persistent backing** - generated package locks are permanently stored in Firestore. This is required to ensure load consistency since this is persisted across instances & deployments. As a result, the package-lock generation performance penalty will only be seen on the _first request for that package version_. To prevent users from ever seeing this latency, requests can be made to generate the package lock before any user needs it.
 * **In-memory cache** - a per-instance least recently used cache is used to improve performance of package lock reads.
 * **Data structure** - package locks are not stored in the same format that they are generated. The initial representation is very large and instead they are converted to a flat object with only package names & versions. This implies that conflicting versions present in `package-lock.json` are not preserved. This results in an ~100x compression vs original `package-lock.json` files (eg 432KB package-lock.json is now 6KB).
 * **Compression** - before storing into the in-memory cache, gzip compression is used to reduce memory and increase the number of items that can be cached. (For the 432KB package-lock.json, this is now 2.2KB).

To use the package lock, the HTML/JS rewriter now inserts the root package string as a query parameter to subsequent requests. These requests can then simply use the same package-lock to resolve versions.
  • Loading branch information
samuelli committed Sep 29, 2018
1 parent 037e019 commit f96c4e5
Show file tree
Hide file tree
Showing 11 changed files with 8,655 additions and 1,722 deletions.
17 changes: 17 additions & 0 deletions demo/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
# Demo service
This is a service which enables demos to be served on webcomponents.org. It
utilizes the unpkg service with support for bare module specifiers.

## Developing
Build this Typescript project:

```
npm run build
```

Run the tests:
```
npm run test
```

## Deployment
This project requires [Cloud Firestore in Native mode](https://cloud.google.com/datastore/docs/firestore-or-datastore#choosing_a_database) when running on Google Cloud Platform. Ensure the project is built.
```
gcloud app deploy demo.yaml --project <your-project-id>
7 changes: 6 additions & 1 deletion demo/demo.yaml
Original file line number Diff line number Diff line change
@@ -1,2 +1,7 @@
runtime: nodejs8
service: demo
service: default

instance_class: F4_1G

automatic_scaling:
min_instances: 1
Loading

0 comments on commit f96c4e5

Please sign in to comment.