Multiple files, modules, require, etc.

[dirk]

We eventually need to get to the issue of how we want to handle multiple files. There are, as I see it, a few approaches to this:

1. Copy Node.js

This is the most brittle approach, however it also ties in easiest with our JavaScript target since all source files can be compiled directly to corresponding JavaScript files.

The big downside to this approach is that require and module.exports provide zero type information, so any uses of values from a required(...) module (internal or external) would have no type information (just a ton of Any types).

We could perhaps develop a hybrid system on top of this. Where Hummingbird files would be loaded with a different keyword/syntax and those files would expose their exported values via a complementary interface. For example:

// Importing Node module and local Hb module
var express = require("express")
var something = import("./something.hb")

let Foo = "bar"
func baz () -> String { return "#yolo" }

// Exporting something in this file
// (compiled down to module.exports)
export Foo, baz

2. Roll our own with different design

Node enforces strict boundaries between files, but we don't necessarily need to follow the same path. I'm very open to a "unifying" system that compiles files together into a unified space instead of the deep tree of modules that Node builds.

The disadvantage of this approach is that we don't always get the same file-to-file compilation path as the former. The latter is that we can decide exactly how we want to design our system free from any bounds imposed by Node.js or any other JavaScript runtime environment. Although this means more work for us and less closely hewing to the JavaScript target, I'm in favor of it for the freedom that it gives us to design our system, APIs, and ability to target other runtimes (PyPy, native, etc.).

A trivial, contrived example off the top of my head:

// Only available when targeting Node.js compilation
var express = require_node("express")
// Loading from Hb package system and local file
var something = import("something")
var other = import("./other")

let Foo = "bar"

export Foo as FOO

Final note: import() vs export

In these examples I use import() with parentheses and export without. The reasoning for this is that, in my mind, importing is an imperative action: you are telling the compiler to do something right now. In contrast, exporting is a declarative action: you are telling the compiler something about this file. This is just a thought and I am in no way attached to these designs.

[dirk]

Notes from conversation with @ryngonzalez:

export as a static declaration is good; for this refer to the design of exports in ES6. Having it static means we can deeply integrated with the type-checker on the export-side. Hopefully we can design import to do the same on the import-side; both sides will then meet in a union of great joy and success.

The importing system of ES6 is however way too confusing, especially with regard to the "defaults" concept and the multi-part import syntax. Instead we're going to design for two specific use cases: importing a whole module and importing a subset of exported bindings from a module.

In pseudo-code these two use cases look like:

# Whole module
import "module"
module.foo(module.bar)

# Subset of bindings
import foo, bar from "module"
foo(bar)

[EvanHahn]

+1 for static modules.

The Wren programming language has thought about this a lot, for what it's worth.

[dirk]

👍 Thanks for the heads up! Had forgotten about Wren, although it's implementation is one of my (if not my) favorite small VMs.

[dirk]

After some reading and thoughts, I'd like to make a proposal for the design of this.

Modules are a compiler feature

If you want a piece of code to talk to another piece of code through the standard language features, then it's going to have to go through the compiler. This will be a core principle that guides further design. If you want to have dynamic code loading then you'll have to go through an external platform "API" (eg. dlopen or <script src></script> or whatever).

Modules interact through linkages

Each module, through its exports, defines a static set of linkages which are exposed to other modules. This paradigm—I believe—will work well with both dynamic (JavaScript) and static targets.

No cyclic dependencies

We may figure this out later, but this is not a headache we need right now.

No load-path trickery

Since modules are a compiler feature you control the load-path through the compiler. End of story. (nb. This is not to say the load-path will be obtuse/unaccessible. The load-path should be a controllable/pluggable feature that allows anyone to write their own package manager.)

In the language

# Importing entire module
import other
# Import entire module with rename
import other as something
# Import specific linkage from module
import linkage from other
# Import renamed linkage and another linkage from module
import linkage as something, another from other

# Exporting
let a = 1
export a
class B { }
export B
var c = 2
export c # Error! Cannot export non-statically-bound variables

[EvanHahn]

I like that the package manager is decoupled from the language in code, but I'd love there to be an officially-recommended (and perhaps officially-maintained) package manager.

[ryngonzalez]

I like the overall approach to the proposal, and I like the given syntax: there's a nice symmetry to having both import and export as declarative links in and out. I also agree with @EvanHahn that the choice to keep load-path as a compiler parameter is a good one to keep things flexible.

It's nice that if you were to want to use HB primarily in a Node environment, and you wanted to intermingle with JavaScript packages, you could still write var blah = require('blah') and it would compile down appropriately. Curious to see how we could target the ES6-style imports/exports, but otherwise, I think this is good. 👍

[dirk]

I think for Node compatibility we'd just expose that in some platform-specific API/module; possibly something like:

let require = Platform.Node.require
let blah    = require("blah")

[ryngonzalez]

Ah ok, that makes sense. There'd be a higher (but worth it) barrier to entry w/r/t porting over code, but the solution above is more elegant and flexible for adding other platform's module-loading in the future.

[dirk]

After discussions with @ryngonzalez, we will be revising the import definition style to the following:

# Importing entire modules
import <foo/bar>
import <something> as other # other.baz()
# Importing specific linkages from modules
import <something> using * # baz()
import <something> using baz # baz()

export will remain the same.