Bug in live testing

[davidanthoff]

When I try the live testing feature on CSTParser, I get this:

Status `C:\Users\david\AppData\Local\Temp\jl_Bd50j1\Project.toml`      
  [00ebfdb7] CSTParser v3.1.1-DEV `C:\Users\david\.julia\dev\CSTParser`
  [0796e94c] Tokenize v0.5.8
  [8dfed614] Test

CSTParser: Error During Test at C:\Users\david\.julia\dev\CSTParser\test\runtests.jl:4
  Got exception outside of a @test
  
Test Summary: | Error  Total
CSTParser     |     1      1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
finish(::Test.DefaultTestSet) at C:\Users\david\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5\Test\src\Test.jl:879
top-level scope at C:\Users\david\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5\Test\src\Test.jl:1125
Revise evaluation error at C:\Users\david\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5\Test\src\Test.jl:1125

Stacktrace:
 [1] methods_by_execution!(::Any, ::VSCodeLiveUnitTesting.Revise.CodeTrackingMethodInfo, ::Dict{Module,Array{Expr,1}}, ::Module, ::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:skip_include,),Tuple{Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\lowered.jl:231
 [2] #eval_with_signatures#90 at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:463 [inlined]
 [3] instantiate_sigs!(::VSCodeLiveUnitTesting.OrderedCollections.OrderedDict{Module,VSCodeLiveUnitTesting.OrderedCollections.OrderedDict{VSCodeLiveUnitTesting.Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}}; mode::Symbol, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:skip_include,),Tuple{Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:471
 [4] track(::Module, ::String; mode::Symbol, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:skip_include,),Tuple{Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:887
 [5] track(::String; kwargs::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:mode, :skip_include),Tuple{Symbol,Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:908
 [6] run_test_loop(::String) at c:\Users\david\source\julia-vscode\scripts\packages\VSCodeLiveUnitTesting\src\VSCodeLiveUnitTesting.jl:48
 [7] #3 at c:\Users\david\source\julia-vscode\scripts\packages\VSCodeLiveUnitTesting\src\VSCodeLiveUnitTesting.jl:104 [inlined]
 [8] withenv(::VSCodeLiveUnitTesting.var"#3#5"{String}, ::Pair{String,String}) at .\env.jl:161
 [9] (::Pkg.Operations.var"#100#104"{String,VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.PackageSpec})() at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1434
 [10] with_temp_env(::Pkg.Operations.var"#100#104"{String,VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.PackageSpec}, ::String) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1348
 [11] (::Pkg.Operations.var"#99#103"{VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1419
 [12] mktempdir(::Pkg.Operations.var"#99#103"{VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at .\file.jl:709
 [13] mktempdir(::Function, ::String) at .\file.jl:707 (repeats 2 times)
 [14] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1386
 [15] live_unit_test(::String, ::String) at c:\Users\david\source\julia-vscode\scripts\packages\VSCodeLiveUnitTesting\src\VSCodeLiveUnitTesting.jl:103
 [16] top-level scope at c:\Users\david\source\julia-vscode\scripts\tasks\task_liveunittesting.jl:5
 [17] include(::Function, ::Module, ::String) at .\Base.jl:380
 [18] include(::Module, ::String) at .\Base.jl:368
 [19] exec_options(::Base.JLOptions) at .\client.jl:296
 [20] _start() at .\client.jl:506

This is what I get for QueryOperators.jl:

Status `C:\Users\david\AppData\Local\Temp\jl_nZFInn\Project.toml`
  [864edb3b] DataStructures v0.17.17
  [e7dc6d0d] DataValues v0.4.13
  [82899510] IteratorInterfaceExtensions v1.0.0
  [2aef5ad7] QueryOperators v0.9.4-DEV `C:\Users\david\.julia\dev\QueryOperators`
  [5e66a065] TableShowUtils v0.2.5
  [8dfed614] Test

┌ Warning: likely failure to return to toplevel, try `ExprSplitter`
└ @ VSCodeLiveUnitTesting.JuliaInterpreter c:\Users\david\source\julia-vscode\scripts\packages\JuliaInterpreter\src\interpret.jl:621
┌ Warning: likely failure to return to toplevel, try `ExprSplitter`
└ @ VSCodeLiveUnitTesting.JuliaInterpreter c:\Users\david\source\julia-vscode\scripts\packages\JuliaInterpreter\src\interpret.jl:621
Core: Error During Test at C:\Users\david\.julia\dev\QueryOperators\test\runtests.jl:7
  Got exception outside of a @test
  
QueryOperators: Error During Test at C:\Users\david\.julia\dev\QueryOperators\test\runtests.jl:5
  Got exception outside of a @test
  
Test Summary:  | Pass  Error  Total
QueryOperators |   31      2     33
  Core         |   31      1     32
ERROR: Some tests did not pass: 31 passed, 0 failed, 2 errored, 0 broken.
finish(::Test.DefaultTestSet) at C:\Users\david\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5\Test\src\Test.jl:879
top-level scope at C:\Users\david\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5\Test\src\Test.jl:1125
Revise evaluation error at C:\Users\david\AppData\Local\Programs\Julia 1.5.3\share\julia\stdlib\v1.5\Test\src\Test.jl:1125

Stacktrace:
 [1] methods_by_execution!(::Any, ::VSCodeLiveUnitTesting.Revise.CodeTrackingMethodInfo, ::Dict{Module,Array{Expr,1}}, ::Module, ::Expr; mode::Symbol, disablebp::Bool, always_rethrow::Bool, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:skip_include,),Tuple{Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\lowered.jl:231
 [2] #eval_with_signatures#90 at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:463 [inlined]
 [3] instantiate_sigs!(::VSCodeLiveUnitTesting.OrderedCollections.OrderedDict{Module,VSCodeLiveUnitTesting.OrderedCollections.OrderedDict{VSCodeLiveUnitTesting.Revise.RelocatableExpr,Union{Nothing, Array{Any,1}}}}; mode::Symbol, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:skip_include,),Tuple{Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:471
 [4] track(::Module, ::String; mode::Symbol, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:skip_include,),Tuple{Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:887
 [5] track(::String; kwargs::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:mode, :skip_include),Tuple{Symbol,Bool}}}) at c:\Users\david\source\julia-vscode\scripts\packages\Revise\src\packagedef.jl:908
 [6] run_test_loop(::String) at c:\Users\david\source\julia-vscode\scripts\packages\VSCodeLiveUnitTesting\src\VSCodeLiveUnitTesting.jl:48
 [7] #3 at c:\Users\david\source\julia-vscode\scripts\packages\VSCodeLiveUnitTesting\src\VSCodeLiveUnitTesting.jl:104 [inlined]
 [8] withenv(::VSCodeLiveUnitTesting.var"#3#5"{String}, ::Pair{String,String}) at .\env.jl:161
 [9] (::Pkg.Operations.var"#100#104"{String,VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.PackageSpec})() at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1434
 [10] with_temp_env(::Pkg.Operations.var"#100#104"{String,VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.PackageSpec}, ::String) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1348
 [11] (::Pkg.Operations.var"#99#103"{VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1419
 [12] mktempdir(::Pkg.Operations.var"#99#103"{VSCodeLiveUnitTesting.var"#3#5"{String},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at .\file.jl:709
 [13] mktempdir(::Function, ::String) at .\file.jl:707 (repeats 2 times)
 [14] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1386
 [15] live_unit_test(::String, ::String) at c:\Users\david\source\julia-vscode\scripts\packages\VSCodeLiveUnitTesting\src\VSCodeLiveUnitTesting.jl:103
 [16] top-level scope at c:\Users\david\source\julia-vscode\scripts\tasks\task_liveunittesting.jl:5
 [17] include(::Function, ::Module, ::String) at .\Base.jl:380
 [18] include(::Module, ::String) at .\Base.jl:368
 [19] exec_options(::Base.JLOptions) at .\client.jl:296
 [20] _start() at .\client.jl:506

I don't get any error for StringBuilders.jl, but that doesn't have an include in the runtests.jl file, so maybe that explains it...

I think we are essentially not using Revise.jl properly here, presumably...

@timholy, any chance you might be able to help? The relevant function is run_test_loop here. The idea is that this function should run test/runtests.jl once, and then rerun it whenever anything changes, either in the tests or in the package or somewhere else. We call this "live testing", so that one can work on the code of a package, and then whenever one saves a file, tests automatically rerun, and fast because we (hopefully) don't have to recompile everything. Does the general approach in that function look sane to you?

[timholy]

After my Friday deadline and a couple days to recover, yes, I might be able to pitch in. Bump this to remind me.

[davidanthoff]

Will do, good luck!

[davidanthoff]

@timholy Bump :)

[timholy]

How do I run it? I tried cloning the repo and then navigated to /tmp/vscode/julia-vscode/scripts/packages/VSCodeLiveUnitTesting/src and did this:

julia> include("VSCodeLiveUnitTesting.jl")
ERROR: LoadError: SystemError: opening file "/tmp/vscode/julia-vscode/scripts/packages/OrderedCollections/src/OrderedCollections.jl": No such file or directory
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:168
  [2] #systemerror#64
    @ ./error.jl:167 [inlined]
  [3] systemerror
    @ ./error.jl:167 [inlined]
  [4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
    @ Base ./iostream.jl:293
  [5] open
    @ ./iostream.jl:282 [inlined]
  [6] open(f::Base.var"#328#329"{String}, args::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./io.jl:328
  [7] open
    @ ./io.jl:328 [inlined]
  [8] read
    @ ./io.jl:434 [inlined]
  [9] _include(mapexpr::Function, mod::Module, _path::String)
    @ Base ./loading.jl:1140
 [10] include(mod::Module, _path::String)
    @ Base ./Base.jl:386
 [11] include(x::String)
    @ Main.VSCodeLiveUnitTesting /tmp/vscode/julia-vscode/scripts/packages/VSCodeLiveUnitTesting/src/VSCodeLiveUnitTesting.jl:1
 [12] top-level scope
    @ /tmp/vscode/julia-vscode/scripts/packages/VSCodeLiveUnitTesting/src/VSCodeLiveUnitTesting.jl:3
 [13] include(fname::String)
    @ Base.MainInclude ./client.jl:444
 [14] top-level scope
    @ REPL[1]:1
in expression starting at /tmp/vscode/julia-vscode/scripts/packages/VSCodeLiveUnitTesting/src/VSCodeLiveUnitTesting.jl:1

Upon investigation, directories like OrderedCollections and Revise are empty.

A "reproducing this for dummies" would be a big help!

[pfitzseb]

I think you just need a git submodule init.

[timholy]

I said dummies 😆. Do you hear me, dummies. Like really, really dumb.

EDIT: here's what I mean

tim@diva:/tmp/vscode/julia-vscode/scripts/packages$ git submodule init
Submodule 'scripts/languageserver/packages/CSTParser' (https://github.com/julia-vscode/CSTParser.jl.git) registered for path 'CSTParser'
Submodule 'scripts/packages/ChromeProfileFormat' (https://github.com/julia-vscode/ChromeProfileFormat.jl.git) registered for path 'ChromeProfileFormat'
Submodule 'scripts/debugger/packages/CodeTracking' (https://github.com/timholy/CodeTracking.jl.git) registered for path 'CodeTracking'
Submodule 'scripts/packages/DebugAdapter' (https://github.com/julia-vscode/DebugAdapter.jl.git) registered for path 'DebugAdapter'
Submodule 'scripts/languageserver/packages/DocumentFormat' (https://github.com/julia-vscode/DocumentFormat.jl) registered for path 'DocumentFormat'
Submodule 'scripts/languageserver/packages/FilePathsBase' (https://github.com/rofinn/FilePathsBase.jl.git) registered for path 'FilePathsBase'
Submodule 'scripts/languageserver/packages/JSON' (https://github.com/JuliaIO/JSON.jl.git) registered for path 'JSON'
Submodule 'scripts/packages/JSONRPC' (https://github.com/julia-vscode/JSONRPC.jl.git) registered for path 'JSONRPC'
Submodule 'scripts/debugger/packages/JuliaInterpreter' (https://github.com/JuliaDebug/JuliaInterpreter.jl.git) registered for path 'JuliaInterpreter'
Submodule 'scripts/languageserver/packages/LanguageServer' (https://github.com/julia-vscode/LanguageServer.jl.git) registered for path 'LanguageServer'
Submodule 'scripts/packages/LoweredCodeUtils' (https://github.com/JuliaDebug/LoweredCodeUtils.jl.git) registered for path 'LoweredCodeUtils'
Submodule 'scripts/packages/OrderedCollections' (https://github.com/JuliaCollections/OrderedCollections.jl.git) registered for path 'OrderedCollections'
Submodule 'scripts/tasks/sysimageenv/PackageCompiler' (https://github.com/JuliaLang/PackageCompiler.jl.git) registered for path 'PackageCompiler'
Submodule 'scripts/packages/Revise' (https://github.com/timholy/Revise.jl.git) registered for path 'Revise'
Submodule 'scripts/languageserver/packages/StaticLint' (https://github.com/julia-vscode/StaticLint.jl) registered for path 'StaticLint'
Submodule 'scripts/languageserver/packages/SymbolServer' (https://github.com/julia-vscode/SymbolServer.jl.git) registered for path 'SymbolServer'
Submodule 'scripts/languageserver/packages/Tokenize' (https://github.com/JuliaLang/Tokenize.jl.git) registered for path 'Tokenize'
Submodule 'scripts/languageserver/packages/URIParser' (https://github.com/JuliaWeb/URIParser.jl.git) registered for path 'URIParser'

but all the directories are still empty.

[timholy]

OK, I just learned a bit about git submodules. Looks like git submodule update installs the dependencies.

[timholy]

This is a path problem. With skip_include = false then Revise just evals the statements with no special handling, and the problem is that if you are executing /some/path/file.jl from a different directory, you're not pre-loading /some/path into the task_local_storage and hence it doesn't get used as the prefix for the next include.

With skip_include=true, Revise detects the include and applies the same path manipulations; it largely works if you just delete your skip_include=false setting. The first time it omits the files included by runtests.jl, but if you then touch that file, then it runs the dependencies too.

Fixing the last issue (not skipping the included files the first time) is probably not too hard, but there are issues that may merit some additional thought. For example, what about a package whose test suite writes new *.jl files and then includes them? Should you watch the written files? If so, how do you avoid endless cycles? Another one: what if PkgA depends on PkgB---if PkgB is devved should A's tests be re-run?