Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download npm imports; parser & resolver improvements #843

Closed
wants to merge 109 commits into from

Conversation

mbostock
Copy link
Member

@mbostock mbostock commented Feb 17, 2024

  • Resolve static npm: imports to a local cache (instead of hot-linking from jsDelivr)
  • Resolve dynamic npm: imports to a local cache (e.g., Mermaid plugins)
  • Resolve npm: specifier versions against the local npm cache (for consistency & performance)
  • Populate the local npm cache during preview
  • Proxy /_npm/ to jsDelivr during preview
  • Add a content hash to file attachments during preview (for live updating)
  • Add a content hash to file attachments during build (for cache-breaking)
  • Add a content hash to stylesheets during preview (for live updating)
  • Add a content hash to stylesheets during build (for cache-breaking)
  • Add a content hash to imported modules during preview (for live updating)
  • Add a content hash to imported modules during build (for cache-breaking)
  • Add a content hash to the search index during build (for cache-breaking)
  • Detect implicit npm: imports based on FileAttachment method or file extension
  • Detect implicit npm: imports based on unbound references (e.g., d3 or Plot)
  • Detect implicit npm: stylesheets (e.g., for Leaflet and Observable Inputs)
  • Fix file references in static HTML
  • Fix file watching during preview
  • Fix incremental update during preview
  • Fix file watching of data loaders
  • Fix resolution of dynamic npm: imports in transpileModule
  • Fix issues downloading Mermaid’s plugins (jsDelivr timeouts)
  • Download (and rewrite) imports from npm during build
  • Download stylesheets from npm during build (e.g., Leaflet)
  • Download global stylesheets (KaTeX, Leaflet, Mapbox) during build
  • Download KaTeX fonts during build
  • Download extra files from npm during build (e.g., wasm bundles)
  • Fix build content hash for files generated by data loaders
  • Fix incremental update when code output is moved

This also includes some fairly major re-architecting of parsing, transpiling, and rendering. Notes to come.

@Fil

This comment was marked as resolved.

@mbostock

This comment was marked as resolved.

@@ -8,7 +8,7 @@ const resultsContainer = document.querySelector("#observablehq-search-results");
const activeClass = "observablehq-link-active";
let currentValue;

const index = await fetch(import.meta.resolve("./minisearch.json"))
const index = await fetch(import.meta.resolve(global.__minisearch))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use the importMetaResolve Rollup plugin to say import.meta.resolve("observablehq:minisearch.json") here, but that’s just polish.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike /_observablehq/client.js which changes only when we update framework, the contents of /_observablehq/search.js change about each time we build the site.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest commit implements import.meta.resolve resolution so you can now say fetch(import.meta.resolve("observablehq:minisearch.json")). 🥳

@Fil Fil mentioned this pull request Mar 1, 2024
Copy link
Contributor

@Fil Fil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great!

}
width: 640,
height: 400,
data: {url: await FileAttachment("gistemp.csv").url(), format: {type: "csv"}},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the file url /_file/lib/gistemp.08e51068.csv is ending in .csv again, so all is well with both versions of the code.

@@ -22,7 +22,7 @@
<label id="observablehq-sidebar-close" for="observablehq-sidebar-toggle"></label>
<li class="observablehq-link"><a href="./">Home</a></li>
</ol>
<div id="observablehq-search" data-root="./"><input type="search" placeholder="Search"></div>
<div id="observablehq-search"><input type="search" placeholder="Search"></div>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@mbostock
Copy link
Member Author

mbostock commented Mar 1, 2024

Seems like GitHub is broken and the latest commit 51dfd9a implementing import.meta.resolve was not showing up, so I manually squash-merged as de4fe09.

@Fil
Copy link
Contributor

Fil commented Mar 1, 2024

import.meta.resolve doesn't seem to work for minisearch.json:

```js
import.meta.resolve("observablehq:minisearch.json");
```

results in:

build node_modules/@observablehq/framework/src/client/minisearch.json →
Unexpected error: Could not resolve entry module "node_modules/@observablehq/framework/src/client/minisearch.json".

UPDATE: when a random file exists under the path src/client/minisearch.json, then the build is correct (points to the correct asset /_observablehq/minisearch.{hash}.json).

@mbostock
Copy link
Member Author

mbostock commented Mar 1, 2024

I tested it and it worked fine, so I dunno @Fil. Can you isolate a reproduction and file a bug if it doesn’t work? Thank you.

@Fil
Copy link
Contributor

Fil commented Mar 2, 2024

filed under #948

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment