Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add reference tests based on examples in docs #22

Closed
hexaeder opened this issue Jun 29, 2021 · 4 comments · Fixed by #70
Closed

add reference tests based on examples in docs #22

hexaeder opened this issue Jun 29, 2021 · 4 comments · Fixed by #70

Comments

@hexaeder
Copy link
Collaborator

as of now, one has to manually check the preview docs for a PR to see wether everything works

I think it would be good to automate this process (reference testing for all of the plots in the docs)

@hexaeder
Copy link
Collaborator Author

hexaeder commented Mar 2, 2022

My current idea for this would be to transform the literate scripts in docs/examples to jl scripts during runtests. For that one could add save commands to the examples which are hidden away in the md export for the documentation. Running the scripts would produce some pngs which could be compared against assets using ReferenceTests.jl or similar.

Additionally, there could be a new literate notebook referencetests.jl where we could put random code snippets to create test files. Even if those examples are not curated nor explained, i think there is value in displaying and testing different features of the package for interested users to scroll through. Stuff like #65 (comment)

@filchristou
Copy link
Collaborator

Currently the test environment is the same as the project environment. I think a first step would be to separate that.
The way I normally do this is with:

cd  test
julia
julia> ]
(@v1.7) pkg> activate .
(test) pkg> dev ..
(test) pkg> add CairoMakie, GLMakie, ...

This way the tester will not need to manually construct the needed environment every time.

Attention
Doing dev .. in the test folder breaks the possibility of calling (GraphMakie) pkg> test from inside the packet. And one need to manually run julia --project test test/runtests.jl. The relevant issue is JuliaLang/Pkg.jl#1585.

@filchristou
Copy link
Collaborator

or is there maybe a reason to go against that ?

@hexaeder
Copy link
Collaborator Author

hexaeder commented Mar 2, 2022

yeah both the nested envionment and the extras section in project have their pros and cons... currently i was using the latter with TestEnv.jl to activate the test environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants