Code structuring discussion

[rjplevin]

As I'm working on the Composite Component feature, I'm also exploring alternative approaches to implement this type of recursive structure conveniently, efficiently, and maintainably in julia.

Several things I'd like to address at some point (after we've got all the julia 1.0 stuff cleaned up):

1. The standard approach to organizing code in julia makes it a little difficult to reason about the code without searching the text to see where things are used. Contributing to this are:

• Mimi.jl has most of the "using ..." statements for the entire pkg. As noted in issue Modify approach to "using xxx" statements #214, I would prefer to have each jl do its own "using ..." statement. See that issue for more info.

• Mimi.jl also exports all public names. I wonder if it would be better to have each jl file export its own "public" names. This keeps the information more local to where it's defined, although a user cannot just look at Mimi.jl to see the public API, but they can run names(Mimi) to retrieve this.

2. I'd like to know when functions and variables are private to files.

• Using a single module (Mimi) for everything means we only have exported and non-exported names, but no way to indicate "private to a file" other than convention. We can use the leading-underscore (_foo) convention from Python, but I'd prefer something more rigorous.

• One option is to use sub-modules. For example, the file build.jl could define within it module build, surrounding all the code in the file, and then using .build. We would explicitly export all names that (i) are intended for use elsewhere in Mimi, (ii) are part of the public API, in which case the names would be re-exported in Mimi.jl. All names defined in Mimi.build that are not exported become effectively private to the file.

3. The best approach to general code organization still isn't obvious. Types have to be defined before they can be referenced, which creates an ordering problem. In the first big overhaul, I organized the code roughly into types (which get loaded first), and large-ish files with definition-related, instance-related, and model-related code. I find that having the types separated from the methods integral to their function is not ideal.

• One approach I'd like to consider is to refactor the code so that each type is defined in its own file, along with functions that require only that type and those loaded before it. This will make for a big PR, but we can ensure that for that PR, nothing changes other than the location of code. Alternatively, we can do this refactoring in stages (e.g., one type at a time) to keep it manageable.

[corakingdon]

RE 1: what's the motivation behind the second bullet? Having all of the exported API in the Mimi.jl file makes sense to me

RE 2: I'm indifferent to this. I guess I haven't encountered the need for knowing when variables are private to a file, but if that weighs as important going forward, then the approach of submodules you described makes sense to me!

RE 3: I really agree with this!! this has been bothering me for a while too.

[rjplevin]

RE 1: what's the motivation behind the second bullet? Having all of the exported API in the Mimi.jl file makes sense to me

It a question of the locality of information. Having each .jl file declare what's public keeps the information more local. However, the way multiple dispatch works, the same name can occur in multiple files, and if it's exported in any of them, they all become public. I don't see a solution to this other than rigorous use of either convention (_foo is private, foo might be public) or submodules.

This question is subsumed by item 2; if we have internal submodules that export only what's public, that would define all functions usable elsewhere within Mimi, while exports in Mimi.jl would define the public API. This seems like the clearest and rigorous approach, and it's simple to implement.