Reorganize code layout

[juli4n]

The current code layout grew fast and organically, and doesn't follow any rule / invariant.

We have most Substrate binaries under /cmd//... although each one structures its packages in a different way. We also have api/, pkg/ internal/ etc.

This proposal tries to introduce some rules so we can have:

• Packages that can be imported by downstream projects in a safe way (no breakages).
• Clean separation of production code vs dev tooling.
• A common place for internal shared packages.
• A common place for internal per component packages.

A good starting point could be:

• cmd/<component>/main.go for the component main packages / entrypoints.
• internal/<component> for component specific specific packages.
• internal for internal shared packages (across components).
• pkg for the shared public packages.
• hack/ dev scripts.
• tools/<tool-name>: dev tools (e.g. benchmarking, load testing).

[thockin]

Reorganizing is probably better sooner than later. Git is pretty good about tracking moves cross rebases, but let's avoid it if we can.

internal/<component> for component specific specific packages.

I think that should be cmd/<component>/ internal -- this represents the most private stuff. Things that a particular component uses but is not eager to share, even with other components. This should probably be the default for most new packages. The bar is relatively low for API design.

Then internal/... is for things that are shared across components, but not to the outside world. This should have a higher bar for API design, testing, and stability.

Then pkg/... is for things that we are okay sharing with the outside world. This should have a very high bar for modularity, API design, testing, and stability. Once we hit a ga release, this comes with some long-term guarantees of stability. This should be used rarely. Most things that would go in here would probably be better as separate repos with their own version numbers.

Under cmd/<component> we should have some structure. I see we have some now, and I haven't thought about whether I like it or not. Some structure is better than no structure. We should try to distinguish things that are part of the deployed system versus things that are tools used in the build or tests. Kubernetes is really bad about that. We should also agree on a convention whether we're going to use the app subdirectory pattern to enable things like integration tests.

Even under hack we should try to keep things in order. It's already starting to become a dumping ground. It would be nice if every fil ein that directory was something that a contributor might want to run by hand under some circumstance. Tools and helpers should be buried further down.

A directory for tools is fine, though we usually stick that under hack. Ben and I discussed module management here, but I forget if we came to a conclusion.

We probably will need a top-level directory for tests and benchmarks that are multi-component integration things.

[juli4n]

Thanks, Tim. Revised proposal:

• cmd/<component>/main.go - component entrypoint
• cmd/<component>/internal/ - component-private packages.
• internal/ - packages shared across components, but not exposed to outside world.
• pkg/ - packages that are ok sharing with downstream projects. High bar for modularity / API design / stability. Should be used with caution / rarely as eventually will imply long-term stability guarrantees.
• hack/ - scripts for dev.
• tests/ - integration tests and benchmark across components.

[BenTheElder]

pkg/ - packages that are ok sharing with downstream projects. High bar for modularity / API design / stability. Should be used with caution / rarely as eventually will imply long-term stability guarrantees.

I think this judgement bar / required context might make this not a "good first issue" FWIW.

[thockin]

I just want to emphasize that pkg/... should be almost empty and those are almost always be candidates for discrete repos. Once we start tagging the repo, those will only be tagged at the same time. No opportunity to fix bugs in between. We don't need to move them to repos yet, but before 1.0 we should evaluate it.

Maybe a small README in each first-level directory, indicating the commitment levels?

[thockin]

If we had OWNERS I would lock down pkg :)