Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions compiler/src/main/exports.rs
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,26 @@ fn read_src(len: usize) -> Result<String, core::str::Utf8Error> {
})
}

/* Pre-fetch feed: each import as `b<TAB>name` (bare, resolve via manifest) or `q<TAB>spec` (quoted URL/path), one per line. */
#[unsafe(no_mangle)]
pub unsafe extern "C" fn extract_imports(len: usize) -> usize {
use crate::modules::packages::{scan_imports, ImportSpec};
let src = match read_src(len) {
Ok(s) => s,
Err(_) => return unsafe { write_out("") },
};
let specs = crate::modules::packages::scan_string_imports(&src);
let joined = specs.join("\n");
unsafe { write_out(&joined) }
let mut buf = alloc::string::String::new();
for spec in scan_imports(&src) {
if !buf.is_empty() { buf.push('\n'); }
let (kind, name) = match &spec {
ImportSpec::Bare(n) => ('b', n),
ImportSpec::Quoted(u) => ('q', u),
};
buf.push(kind);
buf.push('\t');
buf.push_str(name);
}
unsafe { write_out(&buf) }
}

/* Drive one segment of execution; on `Pending*` re-stash the VM into the recycled `PausedRun` box. */
Expand Down
82 changes: 70 additions & 12 deletions compiler/src/modules/packages/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ use alloc::{boxed::Box, string::{String, ToString}, sync::Arc, vec::Vec};

use crate::s;
use crate::modules::vm::types::{HeapPool, Val, VmErr};
use crate::modules::lexer::{lex, Token, TokenType};

pub mod manifest;
pub use manifest::{Manifest, parse_manifest, walk_up_dirs, dir_of, join_relative};
Expand Down Expand Up @@ -101,19 +102,76 @@ pub(crate) fn binding_to_extern(b: &NativeBinding) -> crate::modules::vm::types:
}
}

/* Scans source for quoted from-import specs; WASM host uses results to pre-fetch URLs before compile. */
pub fn scan_string_imports(src: &str) -> Vec<String> {
/* A scanned import: Quoted is a direct URL/path; Bare is a name resolved against the manifest chain. */
#[derive(Debug, Clone, PartialEq)]
pub enum ImportSpec {
Quoted(String),
Bare(String),
}

/* Content between the first quote and its matching close; tolerates string prefixes (r, b, f). Specs carry no escapes, so a raw slice suffices. */
fn unquote(raw: &str) -> String {
let bytes = raw.as_bytes();
let Some(open) = bytes.iter().position(|&c| c == b'"' || c == b'\'') else { return raw.to_string() };
let quote = bytes[open] as char;
match raw[open + 1..].rfind(quote) {
Some(rel) => raw[open + 1..open + 1 + rel].to_string(),
None => raw.to_string(),
}
}

/* Reads the module spec at token `j`: a quoted string or a dotted bare name. Returns (spec, index past it). */
fn read_spec(src: &str, tokens: &[Token], j: usize) -> Option<(ImportSpec, usize)> {
let t = tokens.get(j)?;
match t.kind {
TokenType::String => Some((ImportSpec::Quoted(unquote(&src[t.start..t.end])), j + 1)),
TokenType::Name => {
let mut name = src[t.start..t.end].to_string();
let mut k = j + 1;
// Dotted segments: a.b.c.
while tokens.get(k).map(|x| x.kind) == Some(TokenType::Dot) {
let Some(seg) = tokens.get(k + 1).filter(|s| s.kind == TokenType::Name) else { break };
name.push('.');
name.push_str(&src[seg.start..seg.end]);
k += 2;
}
Some((ImportSpec::Bare(name), k))
}
_ => None,
}
}

/* Every import spec, classified Bare vs Quoted, via the lexer so a `from`/`import` inside a comment or string is never a false hit. */
pub fn scan_imports(src: &str) -> Vec<ImportSpec> {
let (tokens, _errs) = lex(src);
let mut out = Vec::new();
for line in src.lines() {
let t = line.trim_start();
if !t.starts_with("from ") { continue; }
let rest = &t[5..].trim_start();
let bytes = rest.as_bytes();
if bytes.is_empty() || bytes[0] != b'"' { continue; }
let mut end = 1;
while end < bytes.len() && bytes[end] != b'"' { end += 1; }
if end < bytes.len() {
out.push(rest[1..end].to_string());
let mut i = 0;
while i < tokens.len() {
match tokens[i].kind {
TokenType::From => {
if let Some((spec, next)) = read_spec(src, &tokens, i + 1) {
out.push(spec);
// Step past the `import` of this from-statement so it isn't read as a fresh statement.
i = if tokens.get(next).map(|x| x.kind) == Some(TokenType::Import) { next + 1 } else { next };
} else {
i += 1;
}
}
TokenType::Import => {
// `import a, b as c`: comma-separated specs, each with an optional `as` alias.
let mut j = i + 1;
while let Some((spec, next)) = read_spec(src, &tokens, j) {
out.push(spec);
j = next;
if tokens.get(j).map(|x| x.kind) == Some(TokenType::As) {
j += if tokens.get(j + 1).map(|x| x.kind) == Some(TokenType::Name) { 2 } else { 1 };
}
if tokens.get(j).map(|x| x.kind) != Some(TokenType::Comma) { break; }
j += 1;
}
i = j.max(i + 1);
}
_ => i += 1,
}
}
out
Expand Down
2 changes: 1 addition & 1 deletion documentation/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ from dom import query, set_text
set_text(query("#app"), "Hello from Python")
```

`dom` is one of the official [host libraries](/reference/packages#host-libraries-edge-python-host) (`dom`, `network`, `storage` and more), served as JS sources alongside your app; standard `.wasm` packages like [`json`](/reference/packages#json) ship alongside too. See [Official packages](/reference/packages) for the full catalog, and the [runtime README](https://github.com/dylan-sutton-chavez/edge-python/tree/main/runtime) for all `<edge-python>` attributes and the `imports` field for `.py` / `.wasm` modules.
`dom` is one of the official [host libraries](/reference/packages#host-libraries-edge-python-host) (`dom`, `network`, `storage` and more); standard `.wasm` packages like [`json`](/reference/packages#json) sit alongside them. The `packages.json` above declares `dom` explicitly, but the browser runtime also resolves the official packages by bare name with no manifest at all (see [Defaults](/reference/packages#defaults)), fetching each lazily on first import. See [Official packages](/reference/packages) for the full catalog, and the [runtime README](https://github.com/dylan-sutton-chavez/edge-python/tree/main/runtime) for all `<edge-python>` attributes and the `imports` field for `.py` / `.wasm` modules.

## Your first program

Expand Down
4 changes: 2 additions & 2 deletions documentation/reference/imports.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ from utils import *
print(slugify("Hello world"))
```

The names above (`json`, `utils`, `math`) are illustrative, none are built-in. `json` is an [official standard package](/reference/packages#json); the rest stand in for your own modules. Every bare name must be declared in `packages.json` or supplied as a quoted path/URL.
The names above (`json`, `utils`, `math`) are illustrative. `json` is an [official standard package](/reference/packages#json) that the browser runtime resolves by [default](/reference/packages#defaults); `utils` and `math` stand in for your own modules. Apart from the official defaults, every bare name must be declared in `packages.json` or supplied as a quoted path/URL.

## How resolution works

Expand Down Expand Up @@ -88,7 +88,7 @@ Schema:

`from utils import x` resolves to `./lib/utils.py` relative to the entry script; `from math import add` loads `.wasm` per the [wire format](/reference/wasm-abi).

`packages.json` is optional, scripts can use string-form paths directly without project config.
`packages.json` is optional, scripts can use string-form paths directly without project config, and the browser runtime resolves the [official packages](/reference/packages#defaults) by bare name without it.

### Walk-up resolution

Expand Down
39 changes: 18 additions & 21 deletions documentation/reference/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,55 +16,41 @@ Standard packages are host-agnostic (they run wherever WASM runs). Host librarie

## Standard packages (`edge-python-std`)

Language-agnostic `.wasm` plugins over the [WASM module ABI](/reference/wasm-abi). Import by URL or via a `packages.json` alias; the host fetches the `.wasm` and treats its exports as native bindings.
Language-agnostic `.wasm` plugins over the [WASM module ABI](/reference/wasm-abi). Import by bare name (the browser runtime resolves the official ones by default, see [Defaults](#defaults)), by URL, or via a `packages.json` alias; the host fetches the `.wasm` and treats its exports as native bindings.

### `json`

JSON serialization and deserialization, full CPython `json.loads` / `json.dumps` kwargs parity (`object_hook`, `parse_float`, `indent`, `sort_keys`, `ensure_ascii`, `default`, and more).

```python
from "https://std.edgepython.com/json.wasm" import dumps, loads
from json import dumps, loads

data = loads('{"name":"ada","tags":["math","cs"]}')
print(data["name"]) # ada
print(dumps({"k": [1, 2, 3], "ok": True})) # {"k":[1,2,3],"ok":true}
```

Or with a `packages.json` alias so scripts can write the bare name:

```json
{
"imports": {
"json": "https://std.edgepython.com/json.wasm"
}
}
```

```python
from json import dumps, loads
```

Pre-built `.wasm` is published on the [`edge-python-std` releases](https://github.com/dylan-sutton-chavez/edge-python-std). Full API: [`json/README.md`](https://github.com/dylan-sutton-chavez/edge-python-std/tree/main/json).

> **`json` is not built-in.** Examples elsewhere in these docs write `from json import ...` for brevity, but `json` is this external package, you must declare it (alias or URL) like any other module.
> **`json` is an external package, but the browser runtime resolves it by default.** It isn't compiled into `compiler_lib.wasm`, it's this `.wasm` package. In the browser runtime you can write `from json import ...` with no `packages.json` (a built-in [default](#defaults), fetched lazily on first import). Other hosts, or `defaults: false`, need it declared (alias or URL) like any other module.

### `re`

Regular expressions, a CPython `re` subset on a compact backtracking engine. Unicode aware `\d` `\w` `\s` and `(?i)` without shipping Unicode tables, plus capture groups, backreferences, lookahead, and fixed width lookbehind. A step budget raises `RuntimeError` on catastrophic backtracking instead of hanging, so a degrading pattern is reported rather than freezing the worker.

```python
from "https://std.edgepython.com/re.wasm" import search, sub, findall
from re import search, sub, findall

print(search(r'(\d+)-(\d+)', 'order 12-34')) # 12-34
print(sub(r'\s+', '_', 'a b c')) # a_b_c
print(findall(r'\w+', 'one two three')) # ['one', 'two', 'three']
```

Functions: `match`, `search`, `fullmatch`, `findall`, `groups`, `span`, `sub`; flags go inline (`(?i)`, `(?s)`, `(?m)`). The same `packages.json` alias trick lets scripts write the bare `from re import ...`. Pre-built `.wasm` is published on the [`edge-python-std` releases](https://github.com/dylan-sutton-chavez/edge-python-std). Full API: [`re/README.md`](https://github.com/dylan-sutton-chavez/edge-python-std/tree/main/re).
Functions: `match`, `search`, `fullmatch`, `findall`, `groups`, `span`, `sub`; flags go inline (`(?i)`, `(?s)`, `(?m)`). Pre-built `.wasm` is published on the [`edge-python-std` releases](https://github.com/dylan-sutton-chavez/edge-python-std). Full API: [`re/README.md`](https://github.com/dylan-sutton-chavez/edge-python-std/tree/main/re).

## Host libraries (`edge-python-host`)

Plain-JS capabilities that run on the browser's main thread, registered declaratively via the `host` field of [`packages.json`](/reference/imports#packages-json) (with the `<edge-python>` element) or programmatically via `createWorker({ mainThreadModules })`. No `.wasm`, no Rust, no build step. Each call defers to the main thread over `postMessage` (around 0.1 to 0.4 ms); Python sees a synchronous call.
Plain-JS capabilities that run on the browser's main thread, registered declaratively via the `host` field of [`packages.json`](/reference/imports#packages-json) (with the `<edge-python>` element), programmatically via `createWorker({ hostModules })`, or resolved by default with no config at all (see [Defaults](#defaults)). No `.wasm`, no Rust, no build step. Each call defers to the main thread over `postMessage` (around 0.1 to 0.4 ms); Python sees a synchronous call. The ESM loads lazily, the first time a run imports it.

### `dom`

Expand Down Expand Up @@ -126,9 +112,10 @@ Handlers: `time`, `time_ns`, `monotonic`, `monotonic_ns`, `perf_counter`, `perf_

| You have... | Do |
|---|---|
| Any official package, browser runtime | Just `from <name> import ...`, the runtime resolves the official std/host packages by default. Declare it only to pin a different version, or opt out with `defaults: false` |
| A standard `.wasm` package (e.g., `json`) | Quoted URL `from "https://.../json.wasm" import ...`, or a `packages.json` `imports` alias |
| A host library (e.g., `dom`, `network`, `storage`), `<edge-python>` element | Add it to the `host` field of `packages.json` |
| A host library, programmatic `createWorker` | Pass it in `mainThreadModules` |
| A host library, programmatic `createWorker` | Pass its URL in `hostModules` (lazy) or an in-memory factory in `mainThreadModules` (eager) |

```json
{
Expand All @@ -139,6 +126,16 @@ Handlers: `time`, `time_ns`, `monotonic`, `monotonic_ns`, `perf_counter`, `perf_

One manifest drives both directions: `imports` for worker-side `.py` / `.wasm` modules, `host` for main-thread libraries. See [Imports](/reference/imports) for resolution semantics and the full `packages.json` schema, and the [runtime README](https://github.com/dylan-sutton-chavez/edge-python/tree/main/runtime) for `<edge-python>` attributes and `createWorker` options.

### Defaults

The browser runtime ships a built-in base manifest, so the official packages resolve by bare name with **no `packages.json` at all**: the std `.wasm` packages (`json`, `re`) and the host libraries (`dom`, `network`, `storage`, `time`). Three rules:

- **Lazy.** A default is fetched only when a run actually imports it. Unused defaults never hit the network.
- **Overridable.** Your `packages.json` (or `imports` / `hostModules`) wins for the same name, so you can pin a specific version or URL.
- **Opt-out.** Pass `defaults: false` to `createWorker` to disable the base manifest entirely (e.g. offline or non-browser embedders).

Defaults are a convenience of the browser runtime, not the compiler: `compiler_lib.wasm` stays hermetic and resolves bare names only through the manifest the host provides. Non-browser hosts decide their own defaults, if any.

## See also

- [Imports](/reference/imports), import syntax, `packages.json`, integrity verification.
Expand Down
2 changes: 1 addition & 1 deletion documentation/reference/writing-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Browsers run the engine in a Web Worker (no `document`, no `window`). Path C bri

Async handlers (returning a `Promise`) run concurrently when several coroutines call them under `gather`: each result is routed back to the coroutine that issued it, and a rejected handler raises a catchable exception in that one coroutine without disturbing its peers.

No `.wasm`, no Rust, no build step.
Three ways to register: pass the imported object to `mainThreadModules` (eager, shown below); give a URL to `hostModules` or the `packages.json` `host` field, imported lazily the first time a run uses it; or, for the official libraries, rely on the runtime [defaults](/reference/packages#defaults) with no config. No `.wasm`, no Rust, no build step.

### Sketch

Expand Down
Loading
Loading