MetaPkg

A repository of meta-packages and tools to keep them up to date.

There is a common theme among organizations and individuals to have many related packages, all developed and maintained in parallel. Currently, the workflow involves making changes, testing, iterating, and then finally tagging new versions in METADATA.

For any non-static projects (pretty much everything I do), the time between tags is prohibitively long, and people inevitably want to check out master or dev branches to get the latest features. This would normally be fine, but when new features are added, they are frequently due to development in more than one package. The current solutions:

• Don't use the new features, and wait for tagged releases
• Do Pkg.checkout on a package, witness breakage, post an issue or ask in gitter, check out more packages, continue...

This doesn't need to be so difficult, especially when the solution is usually to just run Pkg.checkout on a few repos. Additionally, it becomes annoying to Pkg.free all the related packages properly after a tagged release.

The MetaPkg solution

A meta-package has a list of requirements explaining how to add/remove/checkout/free a group of related packages all at once. This allows Pkg to handle updating in a proper way once your preferred versions are checked out or freed.

As an example, we'll set up a new meta package for the JuliaML ecosystem. To stay flexible, we'll assume the filename is in a constant, fixed location. If unspecified, it uses MetaPkg/requires.

julia 0.5
    LearnBase
    MLDataUtils  master=>tom
    CatViews
    JuliaML/LossFunctions
    JuliaML/Transformations
    JuliaML/PenaltyFunctions
    JuliaML/ObjectiveFunctions
    JuliaML/StochasticOptimization

    tagged:
        StatsBase

There are a few things to note about the format:

• We can have a separate section for each julia version. We'll load the last one where VERSION >= version.
• The tagged section is for packages which we won't checkout... they stay on tagged releases.
• The branch section (label not necessary) will track those packages all on the same branch, doing a Pkg.checkout all to the same branch.
• Use alternate branches by adding one or more metabranch => pkgbranch mappings. Adding a branch name (without the =>) will always check out that branch.

Now that the requirements are defined, we'll load this meta package. Familiar commands are available: add, rm, checkout, and free. Note that you can test out what will happen without actually changing anything by calling MetaPkg.dry_run() at the beginning.

julia> using MetaPkg

julia> MetaPkg.dry_run()

julia> meta_add("MetaLearn")
INFO: Loading MetaSpec from /home/tom/.julia/v0.5/MetaPkg/src/../requires/MetaLearn
INFO: Adding meta package: MetaLearn
INFO: Going to run: Pkg.add("LearnBase")
INFO: Going to run: Pkg.add("MLDataUtils")
INFO: Going to run: Pkg.add("CatViews")
INFO: Going to run: Pkg.clone("git@github.com:JuliaML/LossFunctions.jl")
INFO: Going to run: Pkg.clone("git@github.com:JuliaML/Transformations.jl")
INFO: Going to run: Pkg.clone("git@github.com:JuliaML/PenaltyFunctions.jl")
INFO: Going to run: Pkg.clone("git@github.com:JuliaML/ObjectiveFunctions.jl")
INFO: Going to run: Pkg.clone("git@github.com:JuliaML/StochasticOptimization.jl")

julia> meta_rm("MetaLearn")
INFO: Removing meta package: MetaLearn
INFO: Going to run: Pkg.rm("LearnBase")
INFO: Going to run: Pkg.rm("MLDataUtils")
INFO: Going to run: Pkg.rm("CatViews")
INFO: Going to run: Pkg.rm("LossFunctions")
INFO: Going to run: Pkg.rm("Transformations")
INFO: Going to run: Pkg.rm("PenaltyFunctions")
INFO: Going to run: Pkg.rm("ObjectiveFunctions")
INFO: Going to run: Pkg.rm("StochasticOptimization")

julia> meta_free("MetaLearn")
INFO: Freeing meta package: MetaLearn
INFO: Going to run: Pkg.free("LearnBase")
INFO: Going to run: Pkg.free("MLDataUtils")
INFO: Going to run: Pkg.free("CatViews")
INFO: Going to run: Pkg.checkout("LossFunctions", "master")
INFO: Going to run: Pkg.checkout("Transformations", "master")
INFO: Going to run: Pkg.checkout("PenaltyFunctions", "master")
INFO: Going to run: Pkg.checkout("ObjectiveFunctions", "master")
INFO: Going to run: Pkg.checkout("StochasticOptimization", "master")

julia> meta_checkout("MetaLearn")
INFO: Checking out branch master for meta package: MetaLearn
INFO: Going to run: Pkg.checkout("LearnBase", "master")
INFO: Going to run: Pkg.checkout("MLDataUtils", "tom")
INFO: Going to run: Pkg.checkout("CatViews", "custom")
INFO: Going to run: Pkg.checkout("LossFunctions", "master")
INFO: Going to run: Pkg.checkout("Transformations", "master")
INFO: Going to run: Pkg.checkout("PenaltyFunctions", "master")
INFO: Going to run: Pkg.checkout("ObjectiveFunctions", "master")
INFO: Going to run: Pkg.checkout("StochasticOptimization", "master")

julia> meta_checkout("MetaLearn", "dev")
INFO: Checking out branch dev for meta package: MetaLearn
INFO: Going to run: Pkg.checkout("LearnBase", "dev")
INFO: Going to run: Pkg.checkout("MLDataUtils", "dev")
INFO: Going to run: Pkg.checkout("CatViews", "custom")
INFO: Going to run: Pkg.checkout("LossFunctions", "dev")
INFO: Going to run: Pkg.checkout("Transformations", "dev")
INFO: Going to run: Pkg.checkout("PenaltyFunctions", "dev")
INFO: Going to run: Pkg.checkout("ObjectiveFunctions", "dev")
INFO: Going to run: Pkg.checkout("StochasticOptimization", "dev")