Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download npm imports; parser & resolver improvements #843

Closed
wants to merge 109 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
053273e
only include needed bundles
mbostock Feb 15, 2024
1fcf258
download npm imports
mbostock Feb 17, 2024
da717fd
better content-type; remove sourcemap
mbostock Feb 17, 2024
f5c5df4
only rewrite javascript
mbostock Feb 17, 2024
28c5e61
fix test snapshots
mbostock Feb 17, 2024
bb1b395
fix stylesheet resolution
mbostock Feb 17, 2024
47620f2
npm resolution is now mandatory
mbostock Feb 17, 2024
b0a2e9d
resolve using cache
mbostock Feb 17, 2024
d60ed2b
coalesce concurrent requests
mbostock Feb 17, 2024
ee1865e
docs/.observablehq/cache/_npm
mbostock Feb 18, 2024
68fe32c
rewrite npm imports by parsing
mbostock Feb 18, 2024
5a3d47f
collect global imports
mbostock Feb 18, 2024
b2e726e
Merge branch 'mbostock/prune-bundles' into mbostock/download-npm
mbostock Feb 18, 2024
fda5c4d
files.length
mbostock Feb 18, 2024
d8115e8
implicit file imports
mbostock Feb 18, 2024
a86ce48
getImplicitFileImports tests
mbostock Feb 18, 2024
f25d2e7
Merge branch 'main' into mbostock/download-npm
mbostock Feb 18, 2024
37243ca
avoid port conflict
mbostock Feb 18, 2024
8c65959
rename for clarity
mbostock Feb 18, 2024
3cd842b
Merge branch 'main' into mbostock/download-npm
mbostock Feb 19, 2024
9e20ee0
Merge branch 'main' into mbostock/download-npm
mbostock Feb 20, 2024
3c6512a
fix link rewriting
mbostock Feb 20, 2024
3bd09b6
cleaner path
mbostock Feb 20, 2024
93f71d7
checkpoint build
mbostock Feb 20, 2024
676c90d
partially fix imports
mbostock Feb 21, 2024
2853569
more import resolution fixes
mbostock Feb 21, 2024
d5f36c9
duckdb & sqlite wasm downloads
mbostock Feb 21, 2024
45f3025
only parse .js for transitive imports
mbostock Feb 21, 2024
bcb9205
fix import.meta.resolve
mbostock Feb 21, 2024
f44e5ec
addImplicitDownloads
mbostock Feb 22, 2024
308c4af
comment re. katex stylesheet
mbostock Feb 23, 2024
4601f26
Merge branch 'main' into mbostock/download-npm
mbostock Feb 23, 2024
f3e618e
checkpoint getFiles
mbostock Feb 23, 2024
42392bc
remove findImportDeclarations
mbostock Feb 23, 2024
4e7e07e
resolve path; more tests
mbostock Feb 23, 2024
cd9f14d
more tests
mbostock Feb 23, 2024
a5228ea
assignment tests
mbostock Feb 23, 2024
0a46050
adopt findFiles for parseLocalImports
mbostock Feb 23, 2024
2394a57
remove findFeatures
mbostock Feb 23, 2024
4009fb2
finish findFileAttachments migration
mbostock Feb 23, 2024
c2438af
bound-path createImportResolver
mbostock Feb 23, 2024
7b7ea6d
serve vs. source path
mbostock Feb 23, 2024
2b47d79
more file cleanup
mbostock Feb 24, 2024
24af5c4
fix type import
mbostock Feb 24, 2024
0262b98
simpler import resolution
mbostock Feb 24, 2024
b12f980
root-aware resolution
mbostock Feb 24, 2024
919e76a
more import resolution
mbostock Feb 24, 2024
35efbcb
remove todo
mbostock Feb 24, 2024
8f50421
checkpoint
mbostock Feb 26, 2024
bc9dbb6
checkpoint
mbostock Feb 27, 2024
c0f9e3e
checkpoint
mbostock Feb 27, 2024
8986c73
checkpoint
mbostock Feb 27, 2024
457369c
checkpoint
mbostock Feb 27, 2024
c6dd38d
checkpoint
mbostock Feb 27, 2024
301fcf0
working preview again!
mbostock Feb 27, 2024
73da98f
fix file watching!
mbostock Feb 27, 2024
90c758b
remove test code
mbostock Feb 27, 2024
b836240
remove caveat!
mbostock Feb 27, 2024
d864353
fix data loader file hash
mbostock Feb 27, 2024
3a90984
tidy
mbostock Feb 28, 2024
2d8d0e8
Merge branch 'main' into mbostock/download-npm
mbostock Feb 28, 2024
fb7ae77
more better
mbostock Feb 28, 2024
d17d7c5
minimize diff
mbostock Feb 28, 2024
d3c8e7a
more tests
mbostock Feb 28, 2024
a89ec03
more tests
mbostock Feb 28, 2024
552aba7
fix dissolving head
mbostock Feb 28, 2024
f100a09
more tests
mbostock Feb 28, 2024
8f8834f
Merge branch 'main' into mbostock/download-npm
mbostock Feb 28, 2024
01b8fc9
resolve dynamic imports to jsDelivr
mbostock Feb 28, 2024
352742f
self-host dynamic imports; fix mermaid dist
mbostock Feb 28, 2024
02d105f
comments
mbostock Feb 28, 2024
262b498
comments
mbostock Feb 28, 2024
4a48e92
comments
mbostock Feb 28, 2024
5c0ef75
more tests
mbostock Feb 28, 2024
ab26f0c
more tests
mbostock Feb 28, 2024
5a03ad2
JavaScript parse tests
mbostock Feb 28, 2024
c901851
more tests
mbostock Feb 28, 2024
eb3df9d
fix style coercion
mbostock Feb 28, 2024
9a9a786
build fixes
mbostock Feb 28, 2024
aeb206f
more build fixes
mbostock Feb 28, 2024
60d0c7d
build works?
mbostock Feb 28, 2024
bfa7e51
fix deploy!
mbostock Feb 28, 2024
320630b
build fixes
mbostock Feb 28, 2024
058e72f
fix npm: stylesheets
mbostock Feb 28, 2024
ccf8221
more downloads
mbostock Feb 28, 2024
31de643
fix vega-lite
mbostock Feb 28, 2024
f040e81
fix a couple pages
mbostock Feb 28, 2024
93f0d25
jsdom
mbostock Feb 29, 2024
6cf31fe
simpler tocSelector
mbostock Feb 29, 2024
c65e9fd
move html rewriting to rendering
mbostock Feb 29, 2024
1cdbe61
fix hash, sort of
mbostock Feb 29, 2024
ee4cf53
cleaner
mbostock Feb 29, 2024
1182211
content-hashed file names
mbostock Feb 29, 2024
3feefce
fix build tests
mbostock Feb 29, 2024
dba8eee
tree-shake @observablehq/runtime
mbostock Feb 29, 2024
1df91e4
remove unused import
mbostock Feb 29, 2024
9af1b34
handle missing files
mbostock Feb 29, 2024
c60547e
content-hashed module names
mbostock Feb 29, 2024
e4afd92
content-hashed search index
mbostock Mar 1, 2024
12208f3
content-hashed stylesheets; fix search
mbostock Mar 1, 2024
2687b3c
fix tip </div>
mbostock Mar 1, 2024
796f415
fix for moved cells
mbostock Mar 1, 2024
9694b24
Merge branch 'main' into mbostock/download-npm
mbostock Mar 1, 2024
5f00239
fix </code>
mbostock Mar 1, 2024
5e9497b
fix head: false
mbostock Mar 1, 2024
64c7875
hide searchInput
mbostock Mar 1, 2024
21f4484
keepNames: true
mbostock Mar 1, 2024
9b73d0c
fix shp dbf
mbostock Mar 1, 2024
b4aa2ec
fix canvas max-width
mbostock Mar 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .gitignore
Expand Up @@ -6,6 +6,6 @@ docs/themes.md
docs/theme/*.md
dist/
node_modules/
test/output/*-changed.*
test/output/build/*-changed/
test/output/**/*-changed.*
test/output/build/**/*-changed/
yarn-error.log
3 changes: 0 additions & 3 deletions bin/observable.ts
Expand Up @@ -2,7 +2,6 @@ import {type ParseArgsConfig, parseArgs} from "node:util";
import * as clack from "@clack/prompts";
import {readConfig} from "../src/config.js";
import {CliError} from "../src/error.js";
import {enableNpmVersionResolution, enableRemoteModulePreload} from "../src/javascript/imports.js";
import {faint, link, red} from "../src/tty.js";

const args = process.argv.slice(2);
Expand Down Expand Up @@ -150,8 +149,6 @@ try {
else if (name === "open") values.open = true;
}
const {config, root, host, port, open} = values;
enableNpmVersionResolution(false);
enableRemoteModulePreload(false);
await import("../src/preview.js").then(async (preview) =>
preview.preview({
config: await readConfig(config, root),
Expand Down
2 changes: 1 addition & 1 deletion docs/css/card.md
Expand Up @@ -10,7 +10,7 @@ The `card` class is used to group and delineate content. The `card` classes appl
</div>
```

<div class="tip"><a href="../lib/plot">Observable Plot</a>’s <b>title</b> and <b>subtitle</b> options generate <code>h2</code> and <code>h3</code> elements, respectively, and so will inherit these card styles.<code>
<div class="tip"><a href="../lib/plot">Observable Plot</a>’s <b>title</b> and <b>subtitle</b> options generate <code>h2</code> and <code>h3</code> elements, respectively, and so will inherit these card styles.</div>

Cards can be used on their own, but they most often exist in a [grid](./grid). Cards can contain whatever you like, including text, images, charts, tables, inputs, and more.

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Expand Up @@ -303,7 +303,7 @@ async function json(url) {
const station = await json(`https://api.weather.gov/points/&dollar;{latitude},&dollar;{longitude}`);
const forecast = await json(station.properties.forecastHourly);

process.stdout.write(JSON.stringify(forecast));</pre>
process.stdout.write(JSON.stringify(forecast));</code></pre>

```js
const location = view(Locator([-122.47, 37.8]));
Expand Down
2 changes: 1 addition & 1 deletion docs/inputs/button.md
Expand Up @@ -2,7 +2,7 @@

<a href="https://github.com/observablehq/inputs/blob/main/README.md#button" target="_blank">API</a> · <a href="https://github.com/observablehq/inputs/blob/main/src/button.js" target="_blank">Source</a> · The button input emits an *input* event when you click it. Buttons may be used to trigger the evaluation of cells, say to restart an animation. For example, below is an animation that progressively hides a bar. Clicking the button will restart the animation.

<canvas id="canvas" width="360" height="20" style="max-width: 100%; color: var(--theme-foreground-focus); border: solid 1px var(--theme-foreground);">
<canvas id="canvas" width="360" height="20" style="max-width: 100%; color: var(--theme-foreground-focus); border: solid 1px var(--theme-foreground);"></canvas>

```js echo
const replay = view(Inputs.button("Replay"));
Expand Down
9 changes: 7 additions & 2 deletions docs/inputs/search.md
Expand Up @@ -4,7 +4,12 @@

By default, the query is split into terms separated by spaces; each term is then prefix-matched against the property values (the fields) of each row in the data. Try searching for “gen” below to find Gentoo penguins.

```js echo
```js
const searchInput = Inputs.search(penguins, {placeholder: "Search penguins…"});
const search = view(searchInput);
```

```js run=false
const search = view(Inputs.search(penguins, {placeholder: "Search penguins…"}));
```

Expand All @@ -19,11 +24,11 @@ Inputs.table(search)
```

```js
const input = document.querySelector("input[placeholder='Search penguins…']");
for (const abbr of document.querySelectorAll("abbr")) {
abbr.title = `Search for “${abbr.textContent}”`;
abbr.style.cursor = "default";
abbr.onclick = () => {
const input = searchInput.querySelector("input");
input.value = abbr.textContent;
input.dispatchEvent(new Event("input", {bubbles: true}));
};
Expand Down
1 change: 1 addition & 0 deletions docs/javascript/bar.js
@@ -0,0 +1 @@
export const bar = "bar";
8 changes: 4 additions & 4 deletions docs/javascript/files.md
Expand Up @@ -145,15 +145,15 @@ A common gotcha with JSON is that it has no built-in date type; dates are theref

To display an image, you can use a static image in [Markdown](../markdown) such as `<img src="horse.jpg">` or `![horse](horse.jpg)`. Likewise, you can use a `video` or `audio` element. Per [file-based routing](../routing#files), static references to these files are automatically detected and therefore these files will be included in the built output.

<video src="horse.mp4" autoplay muted loop controls>
<video src="horse.mp4" autoplay muted loop controls></video>

```md
<video src="horse.mp4" autoplay muted loop controls>
```html run=false
<video src="horse.mp4" autoplay muted loop controls></video>
```

If you want to manipulate an image in JavaScript, use `file.image`. For example, below we load an image and invert the RGB channel values.

<canvas id="horse-canvas" width="640" height="512" style="max-width: 100%;">
<canvas id="horse-canvas" width="640" height="512" style="max-width: 100%;"></canvas>

```js echo
const canvas = document.querySelector("#horse-canvas");
Expand Down
1 change: 1 addition & 0 deletions docs/javascript/foo.js
@@ -1 +1,2 @@
export {bar} from "./bar.js"
export const foo = 42;
4 changes: 2 additions & 2 deletions docs/javascript/reactivity.md
Expand Up @@ -38,7 +38,7 @@ const colors = ["#4269d0", "#efb118", "#ff725c", "#6cc5b0"];
const duration = 2000;
```

<canvas id="canvas" width="640" height="30" style="max-width: 100%; height: 30px;">
<canvas id="canvas" width="640" height="30" style="max-width: 100%; height: 30px;"></canvas>

```js echo
const canvas = document.querySelector("#canvas");
Expand All @@ -60,7 +60,7 @@ invalidation.then(() => cancelAnimationFrame(frame));

The `visibility` function returns a promise that resolves when the code block’s display root is visible. This allows you to defer animation or computation until the content scrolls into view. If you missed the animation, try reloading the page and then scrolling down.

<canvas id="canvas2" width="640" height="30" style="max-width: 100%; height: 30px;">
<canvas id="canvas2" width="640" height="30" style="max-width: 100%; height: 30px;"></canvas>

```js echo
await visibility(); // wait until this node is visible
Expand Down
12 changes: 9 additions & 3 deletions docs/lib/shapefile.md
Expand Up @@ -43,7 +43,9 @@ Plot.plot({

Or, streaming to a canvas:

<canvas id="map-canvas" width="960" height="491" style="max-width: 100%;">
<figure style="max-width: 100%;">
<canvas id="map-canvas" width="960" height="491"></canvas>
</figure>

```js echo
const canvas = document.querySelector("#map-canvas");
Expand All @@ -52,8 +54,12 @@ context.fillStyle = getComputedStyle(canvas).color;
context.clearRect(0, 0, canvas.width, canvas.height);

const path = d3.geoPath(d3.geoEquirectangular(), context);
const stream = await FileAttachment("ne_110m_land/ne_110m_land.shp").url();
const source = await shapefile.open(stream);
const source = await shapefile.open(
...(await Promise.all([
FileAttachment("ne_110m_land/ne_110m_land.shp").stream(),
FileAttachment("ne_110m_land/ne_110m_land.dbf").stream()
]))
);

while (true) {
const {done, value} = await source.read();
Expand Down
32 changes: 8 additions & 24 deletions docs/lib/vega-lite.md
Expand Up @@ -25,33 +25,17 @@ Or you can use a [Vega-Lite JSON view specification](https://vega.github.io/vega
```js echo
vl.render({
spec: {
"width": 640,
"height": 400,
"data": {
"url": await FileAttachment("gistemp.csv").url()
},
"mark": "point",
"encoding": {
"x": {
"type": "temporal",
"field": "Date"
},
"y": {
"type": "quantitative",
"field": "Anomaly"
},
"color": {
"type": "quantitative",
"field": "Anomaly",
"scale": {
"range": "diverging",
"reverse": true
}
}
width: 640,
height: 400,
data: {url: await FileAttachment("gistemp.csv").url(), format: {type: "csv"}},
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed because Vega-Lite infers the format type from the file extension (.csv), but doesn’t check for the query string and so is confused by the now-present ?sha=…. It’s better to be explicit about the format anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the file url /_file/lib/gistemp.08e51068.csv is ending in .csv again, so all is well with both versions of the code.

mark: "point",
encoding: {
x: {type: "temporal", field: "Date"},
y: {type: "quantitative", field: "Anomaly"},
color: {type: "quantitative", field: "Anomaly", scale: {range: "diverging", reverse: true}}
}
}
})
```

<div class="tip">When loading data from a file as above, use <a href="../javascript/files"><code>FileAttachment</code></a> so that referenced files are included on <a href="../getting-started#build">build</a>.</div>

2 changes: 1 addition & 1 deletion docs/lib/xlsx.md
Expand Up @@ -60,6 +60,6 @@ Some additional details on values: dates are interpreted as UTC; formula results

If you’d prefer to use [ExcelJS](https://github.com/exceljs/exceljs) directly, you can import it like so:

```js echo
```js run=false
import Excel from "npm:exceljs";
```
28 changes: 0 additions & 28 deletions docs/markdown.md
Expand Up @@ -2,8 +2,6 @@

Markdown in Observable Framework follows the [CommonMark spec](https://spec.commonmark.org/) and is powered by [markdown-it](https://github.com/markdown-it/markdown-it). We also feature [live JavaScript](./javascript) as either [fenced code blocks](./javascript#fenced-code-blocks) (<code>```js</code>) or [inline expressions](./javascript#inline-expressions) (<code>$\{…}</code>), and [HTML in Markdown](#html), and [front matter](#front-matter) for page-level configuration. If you don’t already know Markdown, please see [GitHub’s guide to Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) for an introduction.

<div class="note">Observable Framework currently deviates from CommonMark in how blank lines are handled in HTML; see below. This is a limitation of our parser needed for incremental update during preview.</div>

Here are a few examples of Markdown content to get you started.

## Front matter
Expand Down Expand Up @@ -104,30 +102,4 @@ This produces:
This text is not visible by default.
</details>

In Markdown, blank lines denote separate HTML blocks; be sure to avoid blank lines if you want to treat a chunk of HTML as a single block. For example, write this:

```md
<!-- 👍 one HTML block -->
<ul>
<li>one</li>
<li>two</li>
<li>three</li>
</ul>
```

Don’t write this:

```md
<!-- 👎 three HTML blocks -->
<ul>

<li>one</li>
<li>two</li>
<li>three</li>

</ul>
```

In the latter case, the li elements become top-level and wrapped in a span, rather than children of the ul.

Also see [Hypertext Literal](./lib/htl) for generating dynamic HTML in JavaScript.
7 changes: 5 additions & 2 deletions package.json
Expand Up @@ -55,21 +55,21 @@
"ci-info": "^4.0.0",
"esbuild": "^0.19.8",
"fast-array-diff": "^1.1.0",
"fast-deep-equal": "^3.1.3",
"gray-matter": "^4.0.3",
"he": "^1.2.0",
"highlight.js": "^11.8.0",
"is-docker": "^3.0.0",
"is-wsl": "^3.1.0",
"jsdom": "^24.0.0",
"jszip": "^3.10.1",
"linkedom": "^0.15.6",
"markdown-it": "^13.0.2",
"markdown-it-anchor": "^8.6.7",
"mime": "^3.0.0",
"minisearch": "^6.3.0",
"open": "^9.1.0",
"rollup": "^4.6.0",
"rollup-plugin-esbuild": "^6.1.0",
"semver": "^7.5.4",
"send": "^0.18.0",
"tar-stream": "^3.1.6",
"tsx": "~4.2.1",
Expand All @@ -78,7 +78,9 @@
"ws": "^8.14.2"
},
"devDependencies": {
"@types/d3-array": "^3.2.1",
"@types/he": "^1.2.3",
"@types/jsdom": "^21.1.6",
"@types/markdown-it": "^13.0.2",
"@types/mime": "^3.0.2",
"@types/mocha": "^10.0.2",
Expand All @@ -98,6 +100,7 @@
"eslint-config-prettier": "^9.1.0",
"eslint-import-resolver-typescript": "^3.6.1",
"eslint-plugin-import": "^2.29.0",
"fast-deep-equal": "^3.1.3",
"mocha": "^10.2.0",
"prettier": "^3.0.3 <3.1",
"typescript": "^5.2.2",
Expand Down