feature: engine extensions and quarto api #13701
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Begins the major architectural refactoring to enable external execution engines. Core Architecture: - Split ExecutionEngine into Discovery/Instance pattern - Introduce _discovery flag for new pattern - Create EngineProjectContext as limited ProjectContext subset - Enable loading engines from external URLs - Rename LaunchedExecutionEngine → ExecutionEngineInstance Quarto API Foundation: - Create @quarto/types package with TypeScript definitions - Start implementing QuartoAPI interface - Begin building type system for external engines Co-Authored-By: Claude <noreply@anthropic.com>
Ports Julia engine to the new ExecutionEngineDiscovery/Instance pattern while continuing to build out the Quarto API. Julia Engine: - Port julia to ExecutionEngineDiscovery/Instance interfaces - Explore moving julia out of core (attempted, reverted, restored with override) - Lay groundwork for julia as bundled extension Quarto API Expansion: - Add quarto.jupyter namespace (capabilities, daemon management) - Add quarto.path.* APIs (resource, runtime, dataDir) - Add quarto.format utilities and complete Format type - Add quarto.system APIs for process execution - Add mapped string utilities - Bundle types package for distribution - Pass quarto API to init(), claimsFile, populateCommand - Rename asLaunchedEngine → asEngineInstance throughout Co-Authored-By: Claude <noreply@anthropic.com>
Completes porting of core engines to ExecutionEngineDiscovery pattern and finishes core Quarto API functionality. Jupyter Engine Port: - Port jupyter to ExecutionEngineDiscovery interface (7-part series) - Update kernel management and cell execution for new pattern - Fix uses of legacy ExecutionEngine in jupyter code Knitr Engine Port: - Port knitr to ExecutionEngineDiscovery interface - Update R Markdown engine to work with new pattern API Completion: - Add system.onCleanup and path.dataDir to API - Move _discovery flag to engine implementations - Reorganize types by function for better structure - Clean up type exports and build configuration - Remove quarto.markdown, relocate asYamlText and breakQuartoMd - Port percent.ts to use quarto API - Use mapped types for FormatIdentifier Co-Authored-By: Claude <noreply@anthropic.com>
Completes the architecture migration by cleaning up legacy code and making the check command extensible for external engines. Julia as Bundled Extension: - Use ported julia engine from extension (not core) - Julia now serves as reference implementation for external engines Legacy Code Removal: - Remove legacy ExecutionEngine adapter pattern - Remove deprecated ExecutionEngine interface and _discovery flags - Add quartoRequired version checking for execution engines Check Command Extensibility: - Make quarto check dynamically extensible via ExecutionEngineDiscovery - Move check logic into individual engine implementations - Load external engines in cmd.ts before validation - Add quarto.system.checkRender() API - Fix issue where check was discovering CWD as project API Polish: - Fix jupyter API message function signatures in quarto-types - Add console namespace to Quarto API (spinner, output utilities) - Add inputFilesDir and additional Jupyter methods Co-Authored-By: Claude <noreply@anthropic.com>
Implements tooling and infrastructure for managing engine extensions as git subtrees, enabling them to be maintained in separate repositories while bundling them with Quarto releases. Git Subtree Infrastructure: - Add 'quarto dev-call pull-git-subtree' command to update subtree extensions from their source repositories - Support git subtree directory structure in bundled extensions - Fix pull-git-subtree to search current branch only with debug logging - Remove re-authoring logic from pull-git-subtree Julia Bundled Extension: - Move julia engine to bundled extensions (first bundled engine) - Update tests to account for julia-engine bundled subtree extension Build Fix: - Fix quarto-bld missing V8 experimental regexp engine flag Co-Authored-By: Claude <noreply@anthropic.com>
Final touches to the engine architecture and improvements to Julia engine testing. Architecture Polish: - Fix Windows dynamic import by converting paths to file:// URLs - Make engineCommand dynamic to support external engines - Correct relative path handling - Refactor project initialization and fix julia test cleanup Julia Testing: - Refactor julia tests to use proper Deno test steps - Improve test structure and cleanup Co-Authored-By: Claude <noreply@anthropic.com>
Single-file renders now initialize a ProjectContext with config, enabling
both engine and metadata extensions to work without requiring _quarto.yml.
Key changes:
- Single-file ProjectContext now has config = { project: {} } instead of
undefined, making it consistent with multi-file projects
- Pass full RenderOptions to singleFileProjectContext for extension services
- Export mergeExtensionMetadata() and resolveEngineExtensions() from
project-context.ts for use in single-file path
- Call both functions in singleFileProjectContext() following same order
as regular projects
- Use context.dir consistently for extension discovery
Extension output-dir handling:
- When extensions contribute output-dir metadata, set forceClean to match
the behavior of --output-dir flag, ensuring proper cleanup of the
temporary .quarto directory (addresses #9745, #13625)
- Warn users when extension contributes output-dir, showing the path and
whether cleanup will occur (respects --no-clean flag)
Co-Authored-By: Claude <noreply@anthropic.com>
Completes the external engine ecosystem with user-facing tools and organizational improvements. Git Subtree Organization: - Separate git subtree extensions into extension-subtrees/ directory to distinguish from other bundled extensions - Update subtree extension directory handling and tests - Remove bundled subtree extensions from input to pandoc to avoid duplicate processing Engine Template: - Add 'engine' extension template for quarto create - Include Discovery/Instance pattern boilerplate - Fix hardcoded "example" references in engine template - Improve engine template and error handling - Add cell language prompt for engine configuration - Add engine extension type to basic creation tests Metadata Display: - Filter bundled engines from metadata display output - Suppress expected/noisy engine metadata while preserving user's custom engines Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
gordonwoodhull
force-pushed
the
feature/engine-extension-2
branch
5 times, most recently
from
ad5c773 to
c5f0cd9
Compare
Contributor
Author
|
Wow! Building the package and The most noteworthy one is that external engines depend on import maps, not bundled dependencies, at least in the current iteration. Will discuss internally. This still needs more tests, but maybe it's closer to merge, we'll see. |
gordonwoodhull
changed the title
gordonwoodhull
force-pushed
the
feature/engine-extension-2
branch
from
5f38aca to
8d3147a
Compare
this was a red herring in my investigation, but this is the correct thing to do will squash into the proper commit later.
that reaches out of quarto-cli this caused quarto-latexmk not to build correctly
yet another instance of #9737
this was causing dependencies to blow up where quarto-latexmk was trying to load all execution engines
Co-Authored-By: Claude <noreply@anthropic.com>
Implements MVP of `quarto dev-call build-ts-extension` command that: - Type-checks TypeScript extensions against Quarto API types - Bundles extensions into single JavaScript files using esbuild - Provides default configuration for extensions Key features: - Auto-detects entry point (config, single file, or mod.ts) - Config resolution (user deno.json or default from share/extension-build/) - Type checking with Deno's bundled binary - Bundling with esbuild's bundled binary - --check flag for type-check only Distribution setup: - Static config templates in src/resources/extension-build/ - Copied to share/extension-build/ during packaging - Works in both dev mode and distribution mode Test extension: - Minimal test in tests/smoke/build-ts-extension/ - Implements ExecutionEngineDiscovery interface - Demonstrates type-checking and bundling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…odes Problem: import-map.json pointed to ./quarto-types.d.ts which doesn't exist in dev mode (src/resources/extension-build/). Solution: Follow Lua filter transformation pattern: - Source file in src/resources/ uses dev-mode path - Transform during packaging for distribution Changes: - import-map.json now points to ../../../packages/quarto-types/dist/index.d.ts (works in dev mode from src/resources/extension-build/) - Added updateImportMap() function in prepare-dist.ts that: - Runs after supportingFiles() copies all resources - Transforms @quarto/types path to ./quarto-types.d.ts for distribution - Updates all Deno std versions from src/import_map.json - Loops over all imports dynamically (not hardcoded list) - Updated README.md to document dev vs distribution behavior Result: Default config now works in dev mode without committing generated artifacts to src/resources/. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements --init-config flag that generates a minimal deno.json with
an absolute importMap path pointing to Quarto's bundled configuration.
Features:
- Checks if deno.json already exists and errors with helpful message
- Creates minimal config with compilerOptions and absolute importMap path
- Uses resourcePath() to get correct path in dev and distribution modes
- Exits without building (config generation only)
- Shows helpful customization guidance for users
Example output:
✓ Created deno.json
Import map: /usr/local/share/quarto/extension-build/import-map.json
Customize as needed:
- Add "quartoExtension" section for build options
- Modify "compilerOptions" for type-checking behavior
Generated deno.json points to Quarto's shared import-map.json which
provides @quarto/types and Deno std library imports with correct versions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Remove vendored quarto-types approach and restructure engine template to match the build-ts-extension specification. Changes: - Remove backward compatibility quarto-types copy from prepare-dist.ts - Remove vendoring logic from artifacts/extension.ts - Move TypeScript source from _extensions/ to src/ in engine template - Update _extension.yml to reference .js output instead of .ts - Update imports to use @quarto/types via import map - Add build instructions to README - Fix deno.json to include "deno.ns" in lib array (both default and --init-config) Engine extensions now follow the spec: - Source in src/ - Build to _extensions/<name>/<name>.js using build-ts-extension - Reference types via import map (no vendored copies) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace manual directory scanning with expandGlobSync to find _extension.yml
files, properly supporting nested organization structures like
_extensions/org-name/extension/_extension.yml.
Changes:
- Use expandGlobSync("_extensions/**/_extension.yml") pattern
- Support both flat and nested extension directory structures
- Simplify logic by removing manual directory iteration
- Improve error messages with cleaner relative path display
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Remove Deno std entries from import map since esbuild cannot bundle external URL schemes (jsr:, npm:, https:). Add externals support for marking imports as external (not bundled). Changes to quarto-cli: - Simplify import-map.json: only @quarto/types remains - Simplify updateImportMap(): just path transformation, no version syncing - Add externals?: string[] to quartoExtension interface - Implement --external flag handling in bundle function - Add default externals to deno.json: ["jsr:*", "npm:*", "https:*", "http:*"] Rationale: - esbuild binary only bundles relative file paths - External URL schemes must be resolved at runtime by Deno - Import map now serves single purpose: @quarto/types aliasing - Extension authors use full JSR URLs in source code - Default externals cover all non-bundleable schemes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Switch from esbuild to deno bundle for better import map support and
simpler tooling. deno bundle natively handles JSR/npm/https imports
and does type-checking + bundling in one step.
Changes:
- Replace esbuild with deno bundle in bundle() function
- Remove typeCheck() function (deno bundle does both)
- Restore import map entries for Deno std libraries
- Restore updateImportMap() version syncing in prepare-dist.ts
- Remove externals from quartoExtension interface (not needed)
- Add validateExtensionYml() to warn about path mismatches
- Simplify main action: --check uses deno check, normal build uses deno bundle
Benefits:
- Import maps work natively (no external URL workarounds)
- Extension authors can use bare imports ("path" vs "jsr:@std/path@1.0.8")
- One tool instead of two (simpler, faster)
- Native support for jsr:, npm:, https: imports
- Version consistency with Quarto maintained via import map
- ~60 lines of code removed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds support for specifying the entry point as a command-line argument: quarto dev-call build-ts-extension [entry-point] Entry point resolution priority (highest to lowest): 1. [entry-point] command-line argument 2. quartoExtension.entryPoint in deno.json 3. Single .ts file in src/ directory 4. src/mod.ts if multiple files exist This allows building extensions with multiple TypeScript files without requiring a deno.json just to specify the entry point. Updated error messages to suggest both CLI argument and deno.json options. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Renamed the deno.json configuration section from "quartoExtension" to
"bundle" for better semantic alignment with the underlying deno bundle
command.
Mapping to deno bundle CLI options:
- bundle.entryPoint → [file] positional argument
- bundle.outputFile → --output <output> (✅)
- bundle.minify → --minify (✅)
- bundle.sourcemap → --sourcemap[=<sourcemap>] (✅)
The "bundle" name is more intuitive and accurately reflects that these
are bundling configuration options that map to deno bundle CLI flags.
Updated all references:
- Interface definition: DenoConfig.bundle
- All config.bundle references throughout
- Documentation strings
- Error messages with example JSON
Example deno.json:
{
"bundle": {
"entryPoint": "src/main.ts",
"outputFile": "_extensions/my-ext/main.js",
"minify": true,
"sourcemap": "linked"
}
}
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Refactored output path resolution to support three modes:
1. Just filename (no path separators):
{ "bundle": { "outputFile": "julia-engine.js" } }
→ Infers directory from _extension.yml location
→ Result: _extensions/julia-engine/julia-engine.js
2. Full path (has path separators):
{ "bundle": { "outputFile": "_extensions/custom/out.js" } }
→ Uses path as-is
→ Result: _extensions/custom/out.js
3. Omitted:
{ "bundle": {} }
→ Infers both filename (from entry point) and directory
→ Result: _extensions/julia-engine/julia-engine.js
Implementation changes:
- Split into two focused functions:
- inferFilename(entryPoint): derives "foo.js" from "src/foo.ts"
- inferOutputPath(outputFilename): finds _extension.yml and joins with filename
- Updated bundle() to detect filename-only vs full-path
- Each function has single responsibility, no coupling
This provides flexibility while maintaining convention over configuration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
…m commit 2fb2adf6e git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 2fb2adf6ef7b24272ff76a54ee0b51b5f6962a66
…rces/extension-subtrees/julia-engine'
gordonwoodhull
force-pushed
the
feature/engine-extension-2
branch
from
ab5380c to
3ae0c28
Compare
Contributor
Author
|
Closing due to GH refusing to run tests due to spurious merge conflicts. Will replace |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I still want to add some specific tests, but it's time to make this a draft PR instead of testing directly in CI.
I've addressed the concern about
output-dirbyoutput-dir.forceCleanflag in this case, as we do with--output-dir.I also searched for extensions that use this, and only found three.
Unfortunately, I was not able to squash the commits thematically... way too much churn and merge conflicts.
So they are squashed chronologically into 10 commits, following the plan which added some temporary infrastructure and flags, and then removed it.
The commit of concern, enabling metadata and engine extensions for single files, is separate in 3a813d5, in case we need to revert it.