Accessing an artifact inside a doctest

[mroavi]

I'm using Documenter.doctest(MyPackage; manual=false) inside runtests.jl to run the doctests in my package (as explained here). In one of the doctests, I need to access an artifact. e.g. artifact"myartifact" . I'm getting the following error when I run the tests:

Cannot locate '(Julia)Artifacts.toml' file when attempting to use artifact 'myartifact' in 'Main'

Isn't this currently supported, or do I need to store Artifacts.toml in a specific directory? Storing it in the package root dir, docs dir, or test dir does not seem to work.

[fredrikekre]

Easiest (and most common?) way is to expand the macro in your package and only use a regular binding from the outside, e.g.

myartifact = artifact"myartifact"

in your package, and then use

MyPackage.myartifact

from the outside.

The fact that artifact"..." works in the REPL/Main (I think it does?) is mostly for convenience IIRC, but it doesn't work in arbitrary modules, like the temp ones that are used for doctesting in Documenter.

[mroavi]

Thanks a lot. I will use your solution.

[cossio]

Is there any other way? I have an artifact that is specific to the docs, which I don't think makes a lot of sense to download on every package installation.

[mortenpi]

I'm not very familiar with how artifacts work, but couldn't you have an Artifacts.toml for the docs/Project.toml environment?

[cossio]

@mortenpi The point is that docs/Artifacts.toml doesn't seem to work.

[mortenpi]

What's the error? Is it that artifacts are completely unsupported in non-package environment? Or is it that you can access them in Main / docs/make.jl, but not in doctests?

[cossio]

I see the same error as the OP. It doesn't find the docs/Artifacts.toml in CI nor locally when I run julia --proj make.jl from docs/.

[mortenpi]

Right, so, like in the OP, you can't use the artifacts"" macro within doctests (because they are evaluated in their own modules). But you can use it in make.jl, so one option is to assign the result into a variable there, and then access that variable with Main.my_artifact_name in the doctest.

However, if you do want to use the artifacts"" syntax in the doctest, you could define your own artifacts_str macro in make.jl that calls out to the true artifacts macro, and make it accessible with doctest setup code. I've put together a small example here: https://gist.github.com/mortenpi/9c56cfec1360f0a0f495f79462859317

[cossio]

@mortenpi Thanks. Note however that I'm not trying to find the artifact within a doctest, but within an example Literate file, that becomes a normal documentation page. Not sure if that makes a difference.

However I didn't know artifacts"" worked in make.jl directly, so I'll try that!

[mortenpi]

Literate turns code snippets into at-example blocks, but the same caveats apply.

The only thing that is slightly different is how to bring the macro from make.jl into the scope of the Literate file. You should be able to do something like

import Main: @artifact_str # hide

with # hide there to hide that line from the readers

[mortenpi]

I just want to make one note here: I was looking at the implementation of the artifact"" macro, and I don't actually see why it would fail for doctests. In our case, __source__ should be "none" and pwd() = "../docs/build", so it should be able to find the TOML file as far as I can tell.

I also tried to MWE it with this override in make.jl, but this actually works in a doctest/at-example as far as I can tell:

macro artifact_str(s)
    # Find Artifacts.toml file we're going to load from
    srcfile = @show string(__source__.file)
    @show isinteractive() srcfile isfile(srcfile)
    if ((isinteractive() && startswith(srcfile, "REPL[")) || (!isinteractive() && srcfile == "none")) && !isfile(srcfile)
        srcfile = @show pwd()
    end
    # Sometimes we know the exact path to the Artifacts.toml file, so we can save some lookups
    @show Artifacts.find_artifacts_toml(srcfile)
    :(Artifacts.@artifact_str($s))
end

evaluates in a doctest to

julia> ispath(artifact"socrates")
string(__source__.file) = "none"
isinteractive() = false
srcfile = "none"
isfile(srcfile) = false
pwd() = "/home/mortenpi/juliadocs/artifacts"
Artifacts.find_artifacts_toml(srcfile) = "/home/mortenpi/juliadocs/artifacts/Artifacts.toml"
true

In the same makedocs run, a direct using Artifacts; artifact"socrates" does fail. I'm clearly missing something, but not sure what.

[mroavi]

Let me see if I can reproduce the issue I had originally and provide extra info here.

[mroavi]

I still get the same error. Here is the complete stack trace:

Stack trace

(JunctionTrees) pkg> test
julia>      Testing JunctionTrees
      Status `/tmp/jl_3cVsnE/Project.toml`
⌃ [1520ce14] AbstractTrees v0.4.2
  [e30172f5] Documenter v0.27.23
⌃ [86223c79] Graphs v1.7.2
  [b732b382] JunctionTrees v0.3.3 `~/.julia/dev/JunctionTrees`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_3cVsnE/Manifest.toml`
  [a4c015fc] ANSIColoredPrinters v0.0.1
  [621f4979] AbstractFFTs v1.2.1
⌃ [1520ce14] AbstractTrees v0.4.2
  [79e6a3ab] Adapt v3.4.0
  [ec485272] ArnoldiMethod v0.2.0
⌅ [ab4f0b2a] BFloat16s v0.2.0
  [a9ab73d0] BatchedRoutines v0.2.2
  [7cffe744] BetterExp v0.1.0
  [fa961155] CEnum v0.4.2
  [052768ef] CUDA v3.12.0
⌃ [d360d2e6] ChainRulesCore v1.15.3
  [9e997f8a] ChangesOfVariables v0.1.4
  [861a8166] Combinatorics v1.0.2
⌃ [34da2185] Compat v4.2.0
  [864edb3b] DataStructures v0.18.13
⌃ [ffbed154] DocStringExtensions v0.9.1
  [e30172f5] Documenter v0.27.23
  [e2ba6199] ExprTools v0.1.8
⌃ [5789e2e9] FileIO v1.15.0
  [0c68f7d7] GPUArrays v8.5.0
  [46192b85] GPUArraysCore v0.1.2
⌃ [61eb1bfa] GPUCompiler v0.16.3
⌃ [86223c79] Graphs v1.7.2
  [b5f81e59] IOCapture v0.2.2
  [d25df0c9] Inflate v0.1.3
⌃ [3587e190] InverseFunctions v0.1.7
  [92d709cd] IrrationalConstants v0.1.1
⌃ [033835bb] JLD2 v0.4.22
  [692b3bcd] JLLWrappers v1.4.1
  [682c06a0] JSON v0.21.3
  [b732b382] JunctionTrees v0.3.3 `~/.julia/dev/JunctionTrees`
  [929cbde3] LLVM v4.14.0
⌃ [2ab3a3ac] LogExpFunctions v0.3.18
⌃ [d8e11817] MLStyle v0.4.13
⌃ [1914dd2f] MacroTools v0.5.9
  [626554b9] MetaGraphs v0.7.1
⌃ [ebe7aa44] OMEinsum v0.7.1
⌅ [6f22d1fd] OMEinsumContractionOrders v0.7.1
  [bac558e1] OrderedCollections v1.4.1
⌃ [69de0a69] Parsers v2.3.2
  [21216c6a] Preferences v1.3.0
  [74087812] Random123 v1.6.0
  [e6cf234a] RandomNumbers v1.5.3
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [699a6c99] SimpleTraits v0.9.4
  [276daf66] SpecialFunctions v2.1.7
⌃ [90137ffa] StaticArrays v1.5.6
⌅ [1e83bf80] StaticArraysCore v1.3.0
  [fd094767] Suppressor v0.2.1
⌃ [a759f4b9] TimerOutputs v0.5.21
⌃ [3bb67fe8] TranscodingStreams v0.9.8
  [9d95972d] TupleTools v1.3.0
  [dad2f222] LLVMExtra_jll v0.0.16+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [8ba89e20] Distributed `@stdlib/Distributed`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [1a1011a3] SharedArrays `@stdlib/SharedArrays`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.1 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.2.1 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.20+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [83775a58] Zlib_jll v1.2.12+3 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v5.1.1+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading.
Precompiling project...
  1 dependency successfully precompiled in 6 seconds. 61 already precompiled.
     Testing Running tests...
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
┌ Error: doctest failure in ~/.julia/dev/JunctionTrees/src/junction_tree_algorithm.jl:55-78
│
│ ```jldoctest
│ problem = "Promedus_13"
│ uai_filepath = joinpath(artifact"uai2014", problem * ".uai")
│ uai_evid_filepath = joinpath(artifact"uai2014", problem * ".uai.evid")
│ package_root_dir = pathof(JunctionTrees) |> dirname |> dirname
│ uai_filepath = joinpath(package_root_dir, "docs", "src", "problems", "paskin", "paskin.uai")
│ uai_evid_filepath = joinpath(package_root_dir, "docs", "src", "problems", "paskin", "paskin.uai.evid")
│ algo = compile_algo(
│          uai_filepath,
│          uai_evid_filepath = uai_evid_filepath)
│ eval(algo)
│ obsvars, obsvals = JunctionTrees.read_uai_evid_file(uai_evid_filepath)
│ marginals = run_algo(obsvars, obsvals)
│
│ # output
│
│ 6-element Vector{Factor{Float64, 1}}:
│  Factor{Float64, 1}((1,), [1.0, 0.0, 0.0])
│  Factor{Float64, 1}((2,), [0.0959432982733719, 0.9040567017266281])
│  Factor{Float64, 1}((3,), [0.07863089300137578, 0.9213691069986242])
│  Factor{Float64, 1}((4,), [0.8440129077674895, 0.15598709223251056])
│  Factor{Float64, 1}((5,), [0.9015456486772953, 0.09845435132270475])
│  Factor{Float64, 1}((6,), [0.6118571666785584, 0.3881428333214415])
│ ```
│
│ Subexpression:
│
│ problem = "Promedus_13"
│ uai_filepath = joinpath(artifact"uai2014", problem * ".uai")
│ uai_evid_filepath = joinpath(artifact"uai2014", problem * ".uai.evid")
│ package_root_dir = pathof(JunctionTrees) |> dirname |> dirname
│ uai_filepath = joinpath(package_root_dir, "docs", "src", "problems", "paskin", "paskin.uai")
│ uai_evid_filepath = joinpath(package_root_dir, "docs", "src", "problems", "paskin", "paskin.uai.evid")
│ algo = compile_algo(
│          uai_filepath,
│          uai_evid_filepath = uai_evid_filepath)
│ eval(algo)
│ obsvars, obsvals = JunctionTrees.read_uai_evid_file(uai_evid_filepath)
│ marginals = run_algo(obsvars, obsvals)
│
│ Evaluated output:
│
│ ERROR: LoadError: Cannot locate '(Julia)Artifacts.toml' file when attempting to use artifact 'uai2014' in 'Main'
│ Stacktrace:
│  [1] error(s::String)
│    @ Base ./error.jl:35
│  [2] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any, platform::Any)
│    @ Artifacts ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Artifacts/src/Artifacts.jl:648
│  [3] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any)
│    @ Artifacts ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Artifacts/src/Artifacts.jl:642
│ in expression starting at none:1
│
│ Expected output:
│
│ 6-element Vector{Factor{Float64, 1}}:
│  Factor{Float64, 1}((1,), [1.0, 0.0, 0.0])
│  Factor{Float64, 1}((2,), [0.0959432982733719, 0.9040567017266281])
│  Factor{Float64, 1}((3,), [0.07863089300137578, 0.9213691069986242])
│  Factor{Float64, 1}((4,), [0.8440129077674895, 0.15598709223251056])
│  Factor{Float64, 1}((5,), [0.9015456486772953, 0.09845435132270475])
│  Factor{Float64, 1}((6,), [0.6118571666785584, 0.3881428333214415])
│
│   diff =
│    6-element Vector{Factor{Float64, 1}}:
│     Factor{Float64, 1}((1,), [1.0, 0.0, 0.0])
│     Factor{Float64, 1}((2,), [0.0959432982733719, 0.9040567017266281])
│     Factor{Float64, 1}((3,), [0.07863089300137578, 0.9213691069986242])
│     Factor{Float64, 1}((4,), [0.8440129077674895, 0.15598709223251056])
│     Factor{Float64, 1}((5,), [0.9015456486772953, 0.09845435132270475])
│     Factor{Float64, 1}((6,), [0.6118571666785584, 0.3881428333214415])ERROR: LoadError: Cannot locate '(Julia)Artifacts.toml' file when attempting to use artifact 'uai2014' in 'Main'
│    Stacktrace:
│     [1] error(s::String)
│       @ Base ./error.jl:35
│     [2] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any, platform::Any)
│       @ Artifacts ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Artifacts/src/Artifacts.jl:648
│     [3] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any)
│       @ Artifacts ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Artifacts/src/Artifacts.jl:642
│    in expression starting at none:1
└ @ Documenter.DocTests ~/.julia/dev/JunctionTrees/src/junction_tree_algorithm.jl:55
┌ Error: Doctesting failed
│   exception =
│    `makedocs` encountered a doctest error. Terminating build
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] runner(#unused#::Type{Documenter.Builder.Doctest}, doc::Documenter.Documents.Document)
│        @ Documenter.Builder ~/.julia/packages/Documenter/yf96B/src/Builder.jl:216
│      [3] dispatch(#unused#::Type{Documenter.Builder.DocumentPipeline}, x::Documenter.Documents.Document)
│        @ Documenter.Utilities.Selectors ~/.julia/packages/Documenter/yf96B/src/Utilities/Selectors.jl:170
│      [4] #2
│        @ ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:273 [inlined]
│      [5] cd(f::Documenter.var"#2#3"{Documenter.Documents.Document}, dir::String)
│        @ Base.Filesystem ./file.jl:112
│      [6] makedocs(; debug::Bool, format::Documenter.Writers.HTMLWriter.HTML, kwargs::Base.Pairs{Symbol, Any, NTuple{6, Symbol}, NamedTuple{(:root, :source, :sitename, :doctest, :modules, :doctestfilters), Tuple{String, String, String, Symbol, Vector{Module}, Vector{Regex}}}})
│        @ Documenter ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:272
│      [7] (::Documenter.var"#all_doctests#32"{Bool, Vector{Regex}, Vector{Module}})()
│        @ Documenter ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:941
│      [8] macro expansion
│        @ ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
│      [9] macro expansion
│        @ ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:962 [inlined]
│     [10] macro expansion
│        @ ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
│     [11] doctest(source::Nothing, modules::Vector{Module}; fix::Bool, testset::String, doctestfilters::Vector{Regex})
│        @ Documenter ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:962
│     [12] doctest(package::Module; manual::Bool, testset::Nothing, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
│        @ Documenter ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:897
│     [13] top-level scope
│        @ ~/.julia/dev/JunctionTrees/test/doctests.jl:6
│     [14] include(fname::String)
│        @ Base.MainInclude ./client.jl:476
│     [15] macro expansion
│        @ ~/.julia/dev/JunctionTrees/test/runtests.jl:23 [inlined]
│     [16] macro expansion
│        @ ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
│     [17] top-level scope
│        @ ~/.julia/dev/JunctionTrees/test/runtests.jl:23
│     [18] include(fname::String)
│        @ Base.MainInclude ./client.jl:476
│     [19] top-level scope
│        @ none:6
│     [20] eval
│        @ ./boot.jl:368 [inlined]
│     [21] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:276
│     [22] _start()
│        @ Base ./client.jl:522
└ @ Documenter ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:951
Doctests: JunctionTrees: Test Failed at /home/mroavi/.julia/packages/Documenter/yf96B/src/Documenter.jl:962
  Expression: all_doctests()
Stacktrace:
 [1] macro expansion
   @ ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
 [2] macro expansion
   @ ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:962 [inlined]
 [3] macro expansion
   @ ~/packages/julias/julia-1.8.3/share/julia/stdlib/v1.8/Test/src/Test.jl:1360 [inlined]
 [4] doctest(source::Nothing, modules::Vector{Module}; fix::Bool, testset::String, doctestfilters::Vector{Regex})
   @ Documenter ~/.julia/packages/Documenter/yf96B/src/Documenter.jl:962
Test Summary:             | Fail  Total   Time
doctests                  |    1      1  22.4s
  Doctests: JunctionTrees |    1      1  19.1s
ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /home/mroavi/.julia/dev/JunctionTrees/test/runtests.jl:22
ERROR: Package JunctionTrees errored during testing

runtests.jl:

using Test
using Pkg

Pkg.ensure_artifact_installed("uai2014", "Artifacts.toml")

@testset "doctests" begin
  include("doctests.jl")
end

doctests.jl:

module TestDocs

using Test, Documenter, JunctionTrees

  DocMeta.setdocmeta!(JunctionTrees, :DocTestSetup, :(using JunctionTrees, Artifacts;); recursive=true)
  Documenter.doctest(JunctionTrees; manual=false)

end # module

The Artifacts.toml file is located inside the test dir.
I also made a copy of this file and placed it inside the docs dir.