Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we build examples separately from makedocs? #2309

Closed
glwagner opened this issue Mar 5, 2022 · 3 comments
Closed

Can we build examples separately from makedocs? #2309

glwagner opened this issue Mar 5, 2022 · 3 comments
Labels
documentation 📜 The sacred scrolls

Comments

@glwagner
Copy link
Member

glwagner commented Mar 5, 2022

Our suite of examples is fairly expensive; in recent runs the documentation build exceeds 90 minutes.

If we're able to "build" the examples first, before calling makedocs:

makedocs(bib,
sitename = "Oceananigans.jl",
authors = "Ali Ramadhan, Gregory Wagner, John Marshall, Jean-Michel Campin, Chris Hill",
format = format,
pages = pages,
modules = [Oceananigans],
doctest = true,
strict = true,
clean = true,
checkdocs = :none # Should fix our docstring so we can use checkdocs=:exports with strict=true.
)

we can probably speed up the build. This is crucial now because we need to add new examples both for HydrostaticFreeSurfaceModel and for simulations in complex domains via ImmersedBoundaryGrid. We can also move some examples to the GPU and either speed them up, run them at higher resolution, or both.

What I know now: we provide .md files to makedocs, which then expands code blocks and generates .html:

"Eady turbulence" => "generated/eady_turbulence.md",

"Examples" => example_pages,

pages = pages,

I think one solution is to generate the .html (by running code, which generates images and animations + links in the html) in separate buildkite jobs, and then somehow generate links to the pre-built .html in the "primary" make.jlcall to makedocs before deploydocs.

@glwagner glwagner added the documentation 📜 The sacred scrolls label Mar 5, 2022
@iuryt
Copy link
Collaborator

iuryt commented Mar 9, 2022

I don't know this fits here, but I was thinking about a way to have a separate repository for examples from community that doesn't affect the test running time of the main repo.

The repo could generate a wiki-like page with the examples. We could give a code template for the simulations. For instance, all simulations must inform and check the version of Oceananigans and it's dependencies. Or maybe containing a yml equivalent for Julia project. What do you guys think?

I know this can get messy.. but it could be a nice way to avoid people reinventing the wheel while making their own simulations. With enough time, almost any experiment will have some others similar.

@glwagner
Copy link
Member Author

glwagner commented Mar 9, 2022

I think we'll want a "community repo" for cases eventually.

Something like https://github.com/FluxML/model-zoo if I understand the purpose of that correctly.

For instance, all simulations must inform and check the version of Oceananigans and it's dependencies. Or maybe containing a yml equivalent for Julia project.

Isn't an ordinary Julia environment enough? For the community repo I think either

  1. There is one repo-wide Project.toml and all examples are kept up to date or
  2. Each example has it's own Project.toml.

As I understand the flux model zoo takes approach 1 (this is in principle better, because otherwise the examples grow stale and cease to be useful). However, that requires maintenance and substantial effort.

I think we will still want in-house examples in addition to an external community repo.

@glwagner
Copy link
Member Author

Duplicate of #1053

@glwagner glwagner marked this as a duplicate of #1053 Mar 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation 📜 The sacred scrolls
Projects
None yet
Development

No branches or pull requests

2 participants