Docs

contrib/test_all_models.jl

@@ -21,7 +21,7 @@ packages_to_test = [
     "MimiSNEASY" => ("https://github.com/anthofflab/MimiSNEASY.jl", "master"),
     "MimiFAIR" => ("https://github.com/anthofflab/MimiFAIR.jl", "master"),
     "MimiMAGICC" => ("https://github.com/anthofflab/MimiMAGICC.jl", "master"),
-    "MimiHECTOR" => ("https://github.com/anthofflab/MimiHector.jl", "master")
+    "MimiHector" => ("https://github.com/anthofflab/MimiHector.jl", "master")
 ]
 
 using Pkg

docs/make.jl

@@ -12,7 +12,8 @@ makedocs(
 			"2 Run an Existing Model" => "tutorials/tutorial_2.md",
 			"3 Modify an Existing Model" => "tutorials/tutorial_3.md",
 			"4 Create a Model" => "tutorials/tutorial_4.md",
-			"5 Monte Carlo + Sensitivity Analysis" => "tutorials/tutorial_5.md"
+			"5 Monte Carlo + Sensitivity Analysis" => "tutorials/tutorial_5.md",
+			"6 Create a Model with Composite Components" => "tutorials/tutorial_6.md"
 		],
 		"How-to Guides" => Any[
 			"How-to Guides Intro" => "howto/howto_main.md",
@@ -31,10 +32,14 @@ makedocs(
 		"Reference Guides" => Any[
 			"Reference Guides Intro" => "ref/ref_main.md",
 			"Mimi API" => "ref/ref_API.md",
-			"Structures" => "ref/ref_structures.md", 
+			"Structures: Classes.jl and Types" => "ref/ref_structures_classes_types.md", 
 			"Structures: Definitions" => "ref/ref_structures_definitions.md", 
 			"Structures: Instances" => "ref/ref_structures_instances.md"
 		],
+		"Explanations" => Any[
+			"Explanations Intro" => "explanations/exp_main.md",
+			"Models as Packages" => "explanations/exp_pkgs.md"
+		],
 		"FAQ" => "faq.md",
 	],
 	format = Documenter.HTML(prettyurls = get(ENV, "JULIA_NO_LOCAL_PRETTY_URLS", nothing) === nothing)

docs/src/explanations/exp_main.md

@@ -0,0 +1,9 @@
+# Explanations Introduction
+
+The following reference guides are technical descriptions of the Mimi machinery, organized around the actual code behind Mimi and intended to be information-oriented for those interested in the internals or needing to more deeply understand those aspects in order to carry out their projects. Given the complexity of this code base, these references are not comprehensive, but intended to target some important and relevant aspects of Mimi.
+
+If you find a bug in these reference guides, or have a clarifying question or suggestion, please reach out via Github Issues or our [Mimi Framework forum](https://forum.mimiframework.org).  We welcome your feedback.
+
+## Available Reference Guides
+
+- [Explanations: Models as Packages](@ref) provides a high-level explanation of the practice of organizing models as packages.

docs/src/explanations/exp_pkgs.md

@@ -0,0 +1,49 @@
+## Explanations: Models as Packages
+
+## Models as Packages
+
+[Pkg](https://docs.julialang.org/en/v1/stdlib/Pkg/index.html) (more detail [here](https://julialang.github.io/Pkg.jl/v1/))is Julia's builtin package manager, and handles operations such as installing, updating and removing packages. It is often convenient to set Mimi models up as julia packages using the steps described [here](https://julialang.github.io/Pkg.jl/v1/creating-packages/). "A package is a project with a `name`, `uuid` and version entry in the `Project.toml` file, and a `src/PackageName.jl` file that defines the module `PackageName`. This file is executed when the package is loaded."
+
+Models can be registered or unregistered, as described in the next section, but it is **not necessary** to register a package in order to use the `Pkg` interface.
+
+### Example
+
+The [MimiDICE2016](https://github.com/AlexandrePavlov/MimiDICE2016.jl) model is an unregistered package. As its README instructs, it can be accessed by:
+
+Running the following command at the julia package REPL:
+```julia
+pkg> add https://github.com/AlexandrePavlov/MimiDICE2016.jl
+```
+Now you can use `MimiDICE2016` and its exported API:
+```julia
+using MimiDICE2016
+m = MimiDICE2016.get_model()
+run(m)
+```
+
+## Registries and The Mimi Registry
+
+Packages can be registered in a [Registry](https://julialang.github.io/Pkg.jl/v1/registries/), and "registries contain information about packages, such as available releases and dependencies, and where they can be downloaded. The [General registry](https://github.com/JuliaRegistries/General) is the default one, and is installed automatically".
+
+The Mimi registry is a custom registry maintained by the Mimi development team that colocates several Mimi models in one central registry in the same way julia colates packages in the General registry, where `Mimi` and other packages you commonly may use are located. While the development team maintains this registry and has some basic requirements such as continuous integration tesing (CI) and proper package structure as dictated by julia, they do not claim responsibility or knowledge of the content or quality of the models themselves. 
+
+If you are interested in adding a model to the Mimi Registry, please be in touch with the Mimi development team by opening an [Issue on the Registry](https://github.com/mimiframework/MimiRegistry/issues) and/or a question on the [Mimi forum](https://forum.mimiframework.org) if you do not receive a timely response. We will aim to create a standard guide for this process soon.
+
+### Example
+
+The [MimiDICE2010] model is a registered package in the Mimi Registry, and can be accessed by:
+
+Running the following command at the julia package REPL (only required once):
+```julia
+pkg> registry add https://github.com/mimiframework/MimiRegistry.git
+```
+Followed by adding the package:
+```julia
+pkg> add MimiDICE2010
+```
+Now you can use `MimiDICE2010` and its exported API:
+```julia
+using MimiDICE2010
+m = MimiDICE2010.get_model()
+run(m)
+```

docs/src/howto/howto_1.md

@@ -1,16 +1,20 @@
 # How-to Guide 1: Construct and Run a Model
 
-This how-to guide pairs nicely with Tutorial 4: Create a Model, serving as an abbreviated, less-detailed version and refresher for those with some experience with Mimi. If this is your first time constructing and running a Mimi model, we recommend you start with Tutorial 4, which will give you more detailed step-by step instructions.
+This how-to guide pairs nicely with Tutorial 4: Create a Model and Tutorial 6: Create a Model with Composite Components, and serves as a higher-level version and refresher for those with some experience with Mimi.  If this is your first time constructing and running a Mimi model, we recommend you start with Tutorial 4 (and Tutorial 6 if you are interested in composite components), which will give you more detailed step-by step instructions.
 
 ## Defining Components
 
-Any Mimi model is made up of at least one component, so before you construct a model, you need to create your components.
+Any Mimi model is made up of at least one component, so before you construct a model, you need to create your components. 
 
-A component can have any number of parameters and variables. Parameters are data values that will be provided to the component as input, and variables are values that the component will calculate in the `run_timestep` function when the model is run. The index of a parameter or variable determines the number of dimensions that parameter or variable has. They can be scalar values and have no index, such as parameter 'c' in the example below. They can be one-dimensional, such as the variable 'A' and the parameters 'd' and 'f' below. They can be two dimensional such as variable 'B' and parameter 'e' below. Note that any index other than 'time' must be declared at the top of the component, as shown by `regions = Index()` below.
+Mimi provides two types of components, leaf components and composite components, which generally match intuitively with the classic computer science tree data structure. Note that many existing models are "flat models" with one layer of components, and thus only contain leaf components.
+
+### Leaf Components
+
+A leaf component can have any number of parameters and variables. Parameters are data values that will be provided to the component as input, and variables are values that the component will calculate in the `run_timestep` function when the model is run. The index of a parameter or variable determines the number of dimensions that parameter or variable has. They can be scalar values and have no index, such as parameter 'c' in the example below. They can be one-dimensional, such as the variable 'A' and the parameters 'd' and 'f' below. They can be two dimensional such as variable 'B' and parameter 'e' below. Note that any index other than 'time' must be declared at the top of the component, as shown by `regions = Index()` below.
 
 The user must define a `run_timestep` function for each component. 
 
-We define a component in the following way:
+We define a leaf component in the following way:
 
 ```jldoctest; output = false
 using Mimi
@@ -46,8 +50,94 @@ To access the data in a parameter or to assign a value to a variable, you must u
 
 By default, all parameters and variables defined in the `@defcomp` will be allocated storage as scalars or Arrays of type `Float64.` For a description of other data type options, see How-to Guide 4: Work with Timesteps, Parameters, and Variables 
 
+### Composite Components
+
+Composite components can contain any number of subcomponents, **which can be either leaf components or more composite components**. To the degree possible, composite components are designed to operate in the same way as leaf components, although there are a few necessary differences:
+
+- Leaf components are defined using the macro `@defcomp`, while Composite components are defined using `@defcomposite`. Each macro supports syntax and semantics specific to the type of component.
+
+- Leaf components support user-defined `run_timestep()` functions, whereas composites have a built-in `run_timestep()` function that iterates over its subcomponents and calls their `run_timestep()` function.
+
+A composite component can have any number of parameters and variables, which point to one or more parameters or variables in the composite's subcomponents.  Data all eventually flows through to the leaf components, where calculations are made at runtime and then data is bubbled up into composite components as necessary.
+
+Note that it is not imperative that you explicitly define parameters or variables in a composite component.  It may be desireable for specific use cases, such as ease of access for future connections, future model modification, connecting multiple subcomponent parameters or variables to one higher level component parameter or variable, or parameter conflict resolution (explained below). 
+
+We define a composite component in the following way:
+
+First we will need to have defined some leaf components:
+```julia 
+@defcomp Leaf1 begin
+    par_1_1 = Parameter(index=[time])      
+    var_1_1 = Variable(index=[time])       
+    foo = Parameter()
+
+    function run_timestep(p, v, d, t)
+        v.var_1_1[t] = p.par_1_1[t]
+    end
+end
+
+@defcomp Leaf2 begin
+    par_2_1 = Parameter(index=[time])      
+    par_2_2 = Parameter(index=[time])      
+    var_2_1 = Variable(index=[time])      
+    foo = Parameter()
+
+    function run_timestep(p, v, d, t)
+        v.var_2_1[t] = p.par_2_1[t] + p.foo * p.par_2_2[t]
+    end
+end
+```
+Now we construct a composite component `MyCompositeComponent` which holds the two subcomponents, `Leaf1` and `Leaf2`:
+```julia
+@defcomposite MyCompositeComponent begin
+    Component(Leaf1)
+    Component(Leaf2)
+
+    foo1 = Parameter(Leaf1.foo)
+    foo2 = Parameter(Leaf2.foo)
+
+    var_2_1 = Variable(Leaf2.var_2_1)
+
+    connect(Leaf2.par_2_1, Leaf1.var_1_1)
+    connect(Leaf2.par_2_2, Leaf1.var_1_1)
+end
+```
+
+The `connect` calls are responsible for making internal connections between any two components held by a composite component, similar to `connect_param!` described in the Model section below. 
+
+As mentioned above, conflict resolution refers to cases where two subcomponents have identically named parameters, and thus the user needs to explicitly demonstrate that they are aware of this and create a new external parameter that will point to all subcomponent parameters with that name.  For example, given leaf components `A` and `B`: 
+
+```julia
+@defcomp Leaf1 begin
+    p1 = Parameter()
+    v1 = Variable(index=[time])
+end
+
+@defcomp Leaf2 begin
+    p1 = Parameter()
+end
+```
+The following will fail because you need to resolve the namespace collision of the `p1`'s:
+```julia
+@defcomposite MyCompositeComponent begin
+    Component(Leaf1)
+    Component(Leaf2)
+end
+```
+Fix it with a call to `Parameter` as follows:
+```julia
+@defcomposite MyCompositeComponent begin
+    Component(Leaf1)
+    Component(Leaf2)
+
+    p1 = Parameter(Leaf1.p1, Leaf2.p1)
+end
+```
+
 ## Constructing a Model
 
+Continuing the analogy of a tree data structure, one may consider the Model to be the root, orchestrating the running of all components it contains.
+
 The first step in constructing a model is to set the values for each index of the model. Below is an example for setting the 'time' and 'regions' indexes. The time index expects either a numerical range or an array of numbers.  If a single value is provided, say '100', then that index will be set from 1 to 100. Other indexes can have values of any type.
 
 ```jldoctest; output = false
@@ -84,6 +174,11 @@ set_param!(m, :ComponentName, :ParameterName2, rand(351, 3)) # a two-dimensional
 
 To make an internal connection, the syntax is as follows.  
 
+```julia
+connect_param!(m, :TargetComponent, :ParameterName, :SourceComponent, :VariableName)
+connect_param!(m, :TargetComponent, :ParameterName, :SourceComponent, :VariableName)
+```
+or
 ```julia
 connect_param!(m, :TargetComponent=>:ParameterName, :SourceComponent=>:VariableName)
 connect_param!(m, :TargetComponent=>:ParameterName, :SourceComponent=>:VariableName)
@@ -104,3 +199,107 @@ After all components have been added to your model and all parameters have been
 ```julia
 run(m)
 ```
+
+## Long Example
+
+As a final, lengthier example, below we use the syntax in this tutorial to create and run a toy model with the following structure:
+
+          top
+        /     \
+       A       B
+     /  \     /  \
+    1    2   3    4
+
+```julia
+@defcomp Comp1 begin
+    par_1_1 = Parameter(index=[time])      # external input
+    var_1_1 = Variable(index=[time])       # computed
+    foo = Parameter()
+    function run_timestep(p, v, d, t)
+        v.var_1_1[t] = p.par_1_1[t]
+    end
+end
+
+@defcomp Comp2 begin
+    par_2_1 = Parameter(index=[time])      # connected to Comp1.var_1_1
+    par_2_2 = Parameter(index=[time])      # external input
+    var_2_1 = Variable(index=[time])       # computed
+    foo = Parameter()
+    function run_timestep(p, v, d, t)
+        v.var_2_1[t] = p.par_2_1[t] + p.foo * p.par_2_2[t]
+    end
+end
+
+@defcomp Comp3 begin
+    par_3_1 = Parameter(index=[time])      # connected to Comp2.var_2_1
+    var_3_1 = Variable(index=[time])       # external output
+    foo = Parameter(default=30)
+
+    function run_timestep(p, v, d, t)
+        # @info "Comp3 run_timestep"
+        v.var_3_1[t] = p.par_3_1[t] * 2
+    end
+end
+
+@defcomp Comp4 begin
+    par_4_1 = Parameter(index=[time])      # connected to Comp2.var_2_1
+    var_4_1 = Variable(index=[time])        # external output
+    foo = Parameter(default=300)
+
+    function run_timestep(p, v, d, t)
+        # @info "Comp4 run_timestep"
+        v.var_4_1[t] = p.par_4_1[t] * 2
+    end
+end
+
+@defcomposite A begin
+    Component(Comp1)
+    Component(Comp2)
+
+    foo1 = Parameter(Comp1.foo)
+    foo2 = Parameter(Comp2.foo)
+
+    var_2_1 = Variable(Comp2.var_2_1)
+
+    connect(Comp2.par_2_1, Comp1.var_1_1)
+    connect(Comp2.par_2_2, Comp1.var_1_1)
+end
+
+@defcomposite B begin
+    Component(Comp3)
+    Component(Comp4)
+
+    foo3 = Parameter(Comp3.foo)
+    foo4 = Parameter(Comp4.foo)
+
+    var_3_1 = Variable(Comp3.var_3_1)
+end
+
+@defcomposite top begin
+    Component(A)
+
+    fooA1 = Parameter(A.foo1)
+    fooA2 = Parameter(A.foo2)
+
+    # TBD: component B isn't getting added to mi
+    Component(B)
+    foo3 = Parameter(B.foo3)
+    foo4 = Parameter(B.foo4)
+
+    var_3_1 = Variable(B.var_3_1)
+
+    connect(B.par_3_1, A.var_2_1)
+    connect(B.par_4_1, B.var_3_1)
+end
+
+m = Model()
+set_dimension!(m, :time, 2005:2020)
+add_comp!(m, top, nameof(top))
+set_param!(m, :fooA1, 1)
+set_param!(m, :fooA2, 2)
+set_param!(m, :foo3, 10)
+set_param!(m, :foo4, 20)
+set_param!(m, :par_1_1, collect(1:length(2005:2020)))
+run(m)
+```
+Take a look at what you've created now using `explore(m)`, a peek into what you can learn in How To Guide 2!

docs/src/howto/howto_2.md

@@ -4,7 +4,7 @@ This guide show show to access the numerical values of your results in a usable
 
 ## Accessing Results
 
-After a model has been run, you can access the results (the calculated variable values in each component) in a few different ways.
+After a model has been run, you can access the results (the calculated variable values in each component) in a few different ways. Note that if you have used composite components and wish to access results deeper than the the parameters and variables available in your top-level components you may need the special syntax in the Composite Component Details subsection below.
 
 You can use the `getindex` syntax as follows:
 
@@ -24,6 +24,10 @@ getdataframe(m, :ComponentName=>(:Variable1, :Variable2)) # request multiple var
 getdataframe(m, :Component1=>:Var1, :Component2=>:Var2) # request variables from different components
 ```
 
+### Composite Component Details
+
+[TODO]
+
 ## Plotting and the Explorer UI
 
 Mimi provides support for plotting using [VegaLite](https://github.com/vega/vega-lite) and [VegaLite.jl](https://github.com/fredo-dedup/VegaLite.jl).