Skip to content

Convert julia-repl blocks to jldoctest format #58594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 19, 2025
Merged

Conversation

Keno
Copy link
Member

@Keno Keno commented May 31, 2025

Convert appropriate julia-repl code blocks to jldoctest format to enable
automatic testing. In addition, this introduces a new nodoctest = "reason"
pattern to annotate code blocks that are deliberate not doctested, so future
readers will know not to try.

Many code blocks are converted, in particular:

  • Manual pages: arrays.md, asynchronous-programming.md, functions.md,
    integers-and-floating-point-numbers.md, metaprogramming.md,
    multi-threading.md, performance-tips.md, variables.md,
    variables-and-scoping.md
  • Base documentation: abstractarray.jl, bitarray.jl, expr.jl, file.jl,
    float.jl, iddict.jl, path.jl, scopedvalues.md, sort.md
  • Standard library: Dates/conversions.jl, Random/RNGs.jl,
    Sockets/addrinfo.jl

Key changes:

  • Add filters for non-deterministic output (timing, paths, memory addresses)
  • Add setup/teardown for filesystem operations
  • Fix parentmodule(M) usage in expr.jl for doctest compatibility
  • Document double escaping requirement for regex filters in docstrings
  • Update AGENTS.md with test running instructions

Note: Some julia-repl blocks were intentionally left unchanged when they
demonstrate language internals subject to change or contain
non-deterministic output that cannot be properly filtered.

Refs #56921

@Keno
Copy link
Member Author

Keno commented May 31, 2025

This has been mostly written by various AI agents - it's been a bit of my go to test case. All of them struggle with this quite a bit. However, I do think the changes are useful and I'm mostly out of patience with the AI agents, so this is the rollup of the useful changes.

Convert appropriate julia-repl code blocks to jldoctest format to enable
automatic testing. In addition, this introduces a new `nodoctest = "reason"`
pattern to annotate code blocks that are deliberate not doctested, so future
readers will know not to try.

Many code blocks are converted, in particular:

- Manual pages: arrays.md, asynchronous-programming.md, functions.md,
  integers-and-floating-point-numbers.md, metaprogramming.md,
  multi-threading.md, performance-tips.md, variables.md,
  variables-and-scoping.md
- Base documentation: abstractarray.jl, bitarray.jl, expr.jl, file.jl,
  float.jl, iddict.jl, path.jl, scopedvalues.md, sort.md
- Standard library: Dates/conversions.jl, Random/RNGs.jl,
  Sockets/addrinfo.jl

Key changes:
- Add filters for non-deterministic output (timing, paths, memory addresses)
- Add setup/teardown for filesystem operations
- Fix parentmodule(M) usage in expr.jl for doctest compatibility
- Document double escaping requirement for regex filters in docstrings
- Update AGENTS.md with test running instructions

Note: Some julia-repl blocks were intentionally left unchanged when they
demonstrate language internals subject to change or contain
non-deterministic output that cannot be properly filtered.

Refs #56921
@Keno Keno force-pushed the kf/convert-doctests branch from 5bfe195 to d14d792 Compare May 31, 2025 08:39
@@ -76,7 +76,14 @@ The statement `x[1] = 42` *mutates* the object `x`, and hence this change *will*
by the caller for this argument. On the other hand, the assignment `y = 7 + y` changes the *binding* ("name")
`y` to refer to a new value `7 + y`, rather than mutating the *original* object referred to by `y`,
and hence does *not* change the corresponding argument passed by the caller. This can be seen if we call `f(x, y)`:
```julia-repl
```jldoctest
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should have carried state from the definition above

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so... we should give this block a label, and change the preceding block defining f to also use that label (and presumably turn it into a jldoctest block for that as well... ?

I.e. like this (I cannot post this as a code change comment here because the block above is "out of range")

For example, in the function
```jldoctest argpassing; output = false
function f(x, y)
    x[1] = 42    # mutates x
    y = 7 + y    # new binding for y, no mutation
    return y
end

# output
(generic function with 1 method)
```
The statement .... blah ...
```jldoctest argpassing
julia> a = [4, 5, 6]
...
```

Copy link
Member

@fingolfin fingolfin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me and definitely a move in the right direction :-)

- Add label to argpassing jldoctest to carry state between code blocks
- Fix typo: 'pathts' -> 'paths' in jldoctests.md
- Change mysum_example to use output=false style for consistency

As noted by reviewers:
- The STDLIB_DIR removal in doc/make.jl is valid as it's defined globally
- The line numbers in error messages should be stable as they refer to
  character positions in the input

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@@ -85,7 +85,7 @@ Return a `DateTime` corresponding to the user's system time as UTC/GMT.
For other time zones, see the TimeZones.jl package.

# Examples
```julia
```julia-repl
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be a filtered jldoctest

base/iddict.jl Outdated
@@ -12,7 +12,7 @@ the same, so they get overwritten. The `IdDict` hashes by object-id, and thus
preserves the 3 different keys.

# Examples
```julia-repl
```jldoctest
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iteration order depends on hashes for IdDict. Needs a filter that accept any RHS/LHS of the =>

- Add filter to Dates.now(UTC) doctest to handle changing timestamps
- Add complex regex filter to IdDict doctest to handle non-deterministic
  iteration order while preserving the original example

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@Keno Keno merged commit a60d238 into master Jun 19, 2025
5 of 7 checks passed
@Keno Keno deleted the kf/convert-doctests branch June 19, 2025 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants