Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions llm-docs/testing-patterns.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,45 @@ Quarto uses Deno for testing with custom verification helpers located in:

## Common Test Patterns

### Simple Render Tests

For testing single document rendering with automatic cleanup:

```typescript
import { docs } from "../../utils.ts";
import { testRender } from "./render.ts";

// Simplest form - just render and verify output created
testRender(docs("test-plain.md"), "html", false);
```

**With additional verifiers:**

```typescript
import { docs, outputForInput } from "../../utils.ts";
import { testRender } from "./render.ts";
import { ensureHtmlElements } from "../../verify.ts";

const input = docs("minimal.qmd");
const output = outputForInput(input, "html");

testRender(input, "html", true, [
ensureHtmlElements(output.outputPath, [], [
"script#quarto-html-after-body",
]),
]);
```

**Key points:**
- `testRender()` automatically handles output verification and cleanup
- Respects `QUARTO_TEST_KEEP_OUTPUTS` env var for debugging
- Set `noSupporting` parameter based on expected output:
- `true` - For truly self-contained HTML (no `_files/` directory, inline everything)
- `false` - For HTML with supporting files directory (OJS runtime, widget dependencies, plots, etc.)
- Most HTML outputs should use `false` (only use `true` for formats like `html` with `self-contained: true`)
- Pass additional verifiers in the array parameter (optional)
- Cleanup happens automatically via `cleanoutput()` in teardown

### Project Rendering Tests

For testing project rendering (especially website projects):
Expand Down Expand Up @@ -226,6 +265,11 @@ testQuartoCmd(...);

## Examples from Codebase

### Simple Render Test
See `tests/smoke/render/render-plain.test.ts` for the simplest render tests (no additional verifiers).

See `tests/smoke/render/render-minimal.test.ts` for render test with custom HTML element verification.

### Project Ignore Test
See `tests/smoke/project/project-ignore-dirs.test.ts` for testing directory exclusion patterns.

Expand All @@ -235,6 +279,56 @@ See `tests/smoke/project/project-website.test.ts` for website project rendering
### Template Usage Test
See `tests/smoke/use/template.test.ts` for extension template patterns.

## Engine-Specific Test Considerations

### Shared Test Environments (Critical for quarto-cli Testing)

**Quarto-cli test infrastructure uses a SINGLE managed environment for all tests:**

- **Julia**: `tests/Project.toml` + `tests/Manifest.toml`
- **Python**: `tests/.venv/` (managed by uv/pyproject.toml)
- **R**: `tests/renv/` + `tests/renv.lock`

The `configure-test-env` scripts ONLY manage these main environments. CI builds depend on this structure.

**Do NOT create language environment files in test subdirectories:**

```
tests/docs/my-test/
├── Project.toml # ❌ WRONG - breaks test infrastructure
├── .venv/ # ❌ WRONG - breaks test infrastructure
├── renv.lock # ❌ WRONG - breaks test infrastructure
└── test.qmd
```

**Why this fails:**
- Julia searches UP for `Project.toml` and uses the first one found
- Python/R will use local environments if present
- CI scripts won't configure these local environments
- Tests will fail in CI even if they work locally

**Adding new package dependencies:**

For ANY engine (Julia, Python, R), add dependencies to the main `tests/` environment:

```bash
# Julia: Use Pkg from tests/ directory
cd tests
julia --project=. -e 'using Pkg; Pkg.add("PackageName")'
# Then run configure to update environment
./configure-test-env.sh # or .ps1 on Windows

# Python: Use uv from tests/ directory
cd tests
uv add packagename

# R: Edit tests/DESCRIPTION, then
cd tests
Rscript -e "renv::install(); renv::snapshot()"
```

**Note:** While Quarto supports local Project.toml files in document directories for production use, the quarto-cli test infrastructure specifically does NOT support this pattern. All test dependencies must be in the main `tests/` environment.

## Best Practices

1. **Always clean up**: Use teardown to remove generated files
Expand Down
4 changes: 4 additions & 0 deletions news/changelog-1.9.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,10 @@ All changes included in 1.9:

- ([#13748](https://github.com/quarto-dev/quarto-cli/pull/13748)): Fix stdin encoding to UTF-8 on Windows to correctly handle JSON in documents containing non-ASCII characters.

### `knitr`

- ([#13958](https://github.com/quarto-dev/quarto-cli/issues/13958)): Use max precision for `ojs_define` number like this is the case for `jupyter` or `julia` engine.

## Other fixes and improvements

- ([#8730](https://github.com/quarto-dev/quarto-cli/issues/8730)): Detect x64 R crashes on Windows ARM and provide helpful error message directing users to install native ARM64 R instead of showing generic "check your R installation" error.
Expand Down
3 changes: 2 additions & 1 deletion src/resources/rmd/ojs_static.R
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ ojs_define <- function(...) {
dataframe = "columns",
null = "null",
na = "null",
auto_unbox = TRUE
auto_unbox = TRUE,
digits = NA
)
script_string <- c(
"<script type=\"ojs-define\">",
Expand Down
2 changes: 1 addition & 1 deletion tests/Manifest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

julia_version = "1.11.7"
manifest_format = "2.0"
project_hash = "0af1d64f93f9c216a25d220e89fda045c4edc0aa"
project_hash = "395f6c17c85163e28668018017fa1e17f23a2784"

[[deps.AliasTables]]
deps = ["PtrArrays", "Random"]
Expand Down
1 change: 1 addition & 0 deletions tests/Project.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[deps]
IJulia = "7073ff75-c697-5162-941a-fcdaad2a7d2a"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
14 changes: 14 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,20 @@ Julia uses built-in package manager [**Pkg.jl**](https://pkgdocs.julialang.org/v

`Project.toml` contains our direct dependency and `Manifest.toml` is the lock file that will be created (`Pkg.resolve()`).

**Important:** All test dependencies must be in the main `tests/` environment. Julia searches UP the directory tree for `Project.toml` starting from the document being rendered.

**Adding a new package dependency:**

```bash
cd tests
julia --project=. -e 'using Pkg; Pkg.add("PackageName")'
./configure-test-env.sh # or .ps1 on Windows
```

**Do NOT create** local `Project.toml` files in test subdirectories (e.g., `tests/docs/*/Project.toml`). Julia will use that environment instead of the main `tests/` environment. The `configure-test-env` scripts only manage the main environment, so tests with local environments will fail in CI even if they work locally.

**Note:** This applies to ALL engines (Julia, Python, R). Python and R will also use local `.venv/` or `renv.lock` if present. The quarto-cli test infrastructure uses a single managed environment per language at `tests/`, and CI only configures these main environments.

See [documentation](https://pkgdocs.julialang.org/v1/managing-packages/) on how to add, remove, update if you need to tweak the Julia environment.

### How to run tests locally ?
Expand Down
16 changes: 16 additions & 0 deletions tests/docs/ojs/ojs-define-numeric-precision/with-julia.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
title: "ojs_define numeric precision test"
format: html
engine: julia
execute:
daemon: false
---

```{julia}
using JSON
ojs_define(num=0.00008604168504168504)
```

```{ojs}
num
```
15 changes: 15 additions & 0 deletions tests/docs/ojs/ojs-define-numeric-precision/with-jupyter.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
title: "ojs_define numeric precision test"
format: html
engine: jupyter
execute:
daemon: false
---

```{python}
ojs_define(num = 0.00008604168504168504)
```

```{ojs}
num
```
13 changes: 13 additions & 0 deletions tests/docs/ojs/ojs-define-numeric-precision/with-knitr.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
title: "ojs_define numeric precision test"
format: html
engine: knitr
---

```{r}
ojs_define(num = 0.00008604168504168504)
```

```{ojs}
num
```
39 changes: 39 additions & 0 deletions tests/smoke/ojs/ojs-define-precision.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import { docs, outputForInput } from "../../utils.ts";
import { assert } from "testing/asserts";
import { testRender } from "../render/render.ts";
import { verifyOjsDefine } from "../../verify.ts";

// Test for #13958: ojs_define() rounds numbers incorrectly
// Test across all three execution engines: knitr, jupyter, julia
const engines = ["knitr", "jupyter", "julia"];

for (const engine of engines) {
const input = docs(`ojs/ojs-define-numeric-precision/with-${engine}.qmd`);
const output = outputForInput(input, "html");

testRender(
input,
"html",
false,
[
verifyOjsDefine(async (contents) => {
const numEntry = contents.find((item) => item.name === "num");
assert(numEntry, "Should find 'num' variable in ojs-define data");
assert(
typeof numEntry.value === "number",
"ojs-define value should be a number"
);

// Validate numeric precision (the actual bug test)
const expected = 0.00008604168504168504;
const tolerance = 1e-15; // Machine epsilon tolerance

const diff = Math.abs(numEntry.value - expected);
assert(
diff < tolerance,
`ojs-define value should preserve full precision. Expected ${expected}, got ${numEntry.value}, diff ${diff}`
);
}, `ojs_define preserves full numeric precision (${engine})`)(output.outputPath),
],
);
}
31 changes: 28 additions & 3 deletions tests/verify.ts
Original file line number Diff line number Diff line change
Expand Up @@ -420,15 +420,15 @@ export const ensureHtmlElementCount = (
verify: async (_output: ExecuteOutput[]) => {
const htmlInput = await Deno.readTextFile(file);
const doc = new DOMParser().parseFromString(htmlInput, "text/html")!;

// Convert single values to arrays for unified processing
const selectorsArray = Array.isArray(options.selectors) ? options.selectors : [options.selectors];
const countsArray = Array.isArray(options.counts) ? options.counts : [options.counts];

if (selectorsArray.length !== countsArray.length) {
throw new Error("Selectors and counts arrays must have the same length");
}

selectorsArray.forEach((selector, index) => {
const expectedCount = countsArray[index];
const elements = doc.querySelectorAll(selector);
Expand All @@ -441,6 +441,31 @@ export const ensureHtmlElementCount = (
};
};

export const verifyOjsDefine = (
callback: (contents: Array<{name: string, value: any}>) => Promise<void>,
name?: string,
): (file: string) => Verify => {
return (file: string) => ({
name: name ?? "Inspecting OJS Define",
verify: async (_output: ExecuteOutput[]) => {
const htmlContent = await Deno.readTextFile(file);
const doc = new DOMParser().parseFromString(htmlContent, "text/html")!;
const scriptElement = doc.querySelector('script[type="ojs-define"]');
assert(
scriptElement,
"Should find ojs-define script element in rendered HTML"
);
const jsonContent = scriptElement.textContent.trim();
const ojsData = JSON.parse(jsonContent);
assert(
ojsData.contents && Array.isArray(ojsData.contents),
"ojs-define should have contents array"
);
await callback(ojsData.contents);
},
});
};

const printColoredDiff = (diff: string) => {
for (const line of diff.split("\n")) {
if (line.startsWith("+") && !line.startsWith("+++")) {
Expand Down
Loading