Proposal: Merge website repo into rollup repo

[lukastaegert]

At the moment, documentation is maintained in a separate repo. This leads to documentation constantly running out of date and makes it very hard to find documentation for old versions of rollup.

Proposal

We merge the complete website into a sub-folder of the rollup repository. That way we can add (and should add!) documentation changes to the very same PRs that implement or change features.

• This will also make reviews easier as the reviewer can first read through the intended changes in the documentation before diving into the code.
• Other people looking at a PR might also find the added documentation helpful
• We can "time travel" through documentation together with the accompanying rollup implementation

As documentation is a big issue for many, I would hope to have this in place for the 1.0.0.

This could also be the first step towards a mono-repo containing the core plugins of Rollup but this would probably require more work and discussion.

Necessary prerequisites

At the moment, the documentation is automatically released on merge to master. Thus the access token does not need to be stored locally. In order to maintain this process, we need to implement something like #2268 first.

Thus the documentation is automatically released together with the new version of Rollup.

[guybedford]

Agreed it would be preferable to have docs PRs part of feature PRs as an expectation by having them in the same repo.

I'd prefer not to merge the whole website though - could we just merge the docs folder then keep that in sync with the site through the site scripts?

[lukastaegert]

could we just merge the docs folder then keep that in sync with the site through the site scripts

Why? That sounds unnecessarily complicated, difficult to maintain and difficult to understand for others. I think a mono repo is a worthy goal and the website which has very few issues and open PRs could be a worthy first contender to be merged.

[lukastaegert]

Also this could mean if the structure of the docs folder changes, you will again have to figure out which version of rollup needs to be used with which version of the website to have working documentation.

[guybedford]

It creates a separation between the internals of the website itself, and the markdown files which form the website documentation content.

This is what NodeJS does - https://github.com/nodejs/node/tree/master/doc is used to generate https://nodejs.org/dist/latest-v10.x/docs/api/ with all versions available in parallel.

[lukastaegert]

At the moment, the structure of the website is not very stable so I would not like to separate these two. But it might be worth checking out if the NodeJS approach could be adapted for RollupJS even if everything is in the same repo.

I would regard browsing old documentation as an edge use case anyway.

Something along these lines should be easy to implement and even preserve the full history: https://stackoverflow.com/a/14470212

[lukastaegert]

Also, the REPL needs to be updated as Rollup receives new features or changes its API. This could be another argument for inlining the website.

[guybedford]

I'd prefer to keep the "project and docs" separate from the details of the "website", but it really gets into the discussion of monorepos which as you may have gathered I'm generally personally against as I find them a pain to work with (how do I run this thing? how do I build this thing?). It's a perfectly valid norm though.

[lukastaegert]

I understand monorepos can be intimidating when done wrong. But beyond the possibly intimidating effect, there are only advantages to this. And the more of the core plugins we put into the repo (not as part of this proposal but something I hope to achieve in the future), the more reliable our releases will get. Just imagine that it is impossible to publish rollup or one of its plugins unless all tests of rollup and all plugins have run together with the latest versions of each other. Which would be really easy in a monorepo but a pain to set up and get right with separate repos. This would have prevented quite a few issues with plugins we had in the past. And I am not the first suggesting to go that way.

So, as I really understand your reservations concerning a monorepo with regard to manageability, allow me to sketch a proposal that is really simple and transparent:

1. We move all of rollup as it is into a folder "rollup"
2. The website goes into a separate folder "rollupjs.org" next to the rollup folder
3. In the top folder, there is only a simple readme describing this simple setup and linking to the readmes in the sub folders; also, the travis/appveyor configs go here
4. No additional tooling for now, no lerna or who knows what

And that's it. You want to work on rollup? Just go into the rollup folder and work as before. The same goes for the website.

To keep this setup simple, we try to avoid interdependencies as much as possible.

What do you think? Can I sweeten you this approach?

[guybedford]

Which would be really easy in a monorepo but a pain to set up and get right with separate repos.

This seems to be a common perception, which I don't actually agree with. Just because npm link has been incredibly flaky in the past doesn't mean these workflows can't be made to be easy.

I do also think multi-version documentation will be important in future as we start making major releases now, so worry about locking the website to a single documentation version.

How about a rollup/all repo that contains the website and rollup as git submodules, where the site has some scripts to auto-detect ../rollup handling?

[lukastaegert]

I must admit I was not too familiar with git submodules so I read most of the documentation here: https://git-scm.com/book/en/v2/Git-Tools-Submodules
and yes, they would solve some of the issues. But not all of them. My goal here is to unite code and documentation. And as the REPL part of the website also depends on the core of rollup, I would love to have the code of the website included as well.

I want people to make combined PRs to both the website and rollup itself. And in the future maybe even combined PRs to rollup and its plugins. Submodules would make this possible but it looks like they are a pain to work with and hard to explain to outsiders. Even Github agrees here in their own documentation of submodules: https://blog.github.com/2016-02-01-working-with-submodules/

And if we want to have multiple versions of the documentation, it would be even easier to do this in a monorepo. Each time we release a new version, the current version of the documentation could for instance be copied into an appropriately named folder and committed. Very easy to set up even in a cross platform way. And since the documentation would automatically have the same commit tags as rollup itself, it would be very easy to find the documentation for older versions.

On the contrary with separate repos, you always have to manage the boundaries between the repos and the more boundaries there are across which you want to reach with scripts, npm link etc., the more pain there will be. And I know this kind of pain from work first hand where we have a lot of dependencies we have to keep separate because of guidelines of the customer and this HURTS.

I have not seen any substantial arguments against a monorepo from you (except the documentation argument which, as noted above, I rather see as an argument for a monorepo) but I have the impression that you have a bad feeling about this.

So it would be good to know where this feeling comes from. Maybe you had to work with a some monorepos that were a pain to work with. So please share and let's learn from that and not repeat their mistakes.

I could also just set something up and we can continue discussing this over a PR (yes, I really want this).

[guybedford]

I have not seen any substantial arguments against a monorepo from you (except the documentation argument which, as noted above, I rather see as an argument for a monorepo) but I have the impression that you have a bad feeling about this.

I've never worked with a monorepo that wasn't a pain to work with :) The Babel repo is always confusing to get into, and I find always inhibits my willingness to create contributions.

Monorepos in C++ codebases make a lot of sense to easily run builds, but in the world of JS where modular boundaries are much better handled npm link can handle these cases.

And I know this kind of pain from work first hand where we have a lot of dependencies we have to keep separate because of guidelines of the customer and this HURTS.

Can you detail what frictions you're finding?

And if we want to have multiple versions of the documentation, it would be even easier to do this in a monorepo. Each time we release a new version, the current version of the documentation could for instance be copied into an appropriately named folder and committed.

I really don't like bloat in repos. Separate repos allow eg a plugin to be deprecated and forgotten, while a monorepo means the plugin just sits there and gathers dust. If Rollup goes this route, there will exist multiple folders that are badly maintained and not understood by anyone which means that the "Rollup" repo itself is now badly maintained and not understood by anyone :) Also repos are a natural distinction of ownership, which is an incredibly useful part of the goal of writing a tool that does one thing well.

I think it's really important not to jump to solutions, and instead clearly consider what the problems are, and then assess solutions from there. It sounds to me like the problem is keeping code and documentation updates together - and we should be discussing ensuring the process for this first and foremost.

[lukastaegert]

Ok, I see your argument with deprecated plugins and in favour of a "clean" history and the one-way nature of merging plugins to rollup, I agree to leave them separate.

I think it's really important not to jump to solutions, and instead clearly consider what the problems are, and then assess solutions from there

Which brings me back to the original issue at the top:

I would love to have a union of documentation and code. I.e. I want pull-requests for code changes to contain the accompanying documentation changes and I want this to become a policy

The minimum requirement to make this work is to merge the doc folder. If this is as far as you would be willing to go, then I will accept that. Let me make the case for merging the website as well, though (in this scenario, I would leave rollup in place and put the website into a sub-folder):

• In contrast to e.g. plugins, no-one except us depends on the website repo. Thus we can not only safely deprecate it and put it into rollup's repo, we can just as well separate it again at a later time if we prefer that.
• If we merge the website, it would be easy to add a way to use it as a debugging tool for the currently developed version of rollup without any additional setup.
• If we require documentation changes to be part of rollup PRs, having the website merged would also make it easy to preview documentation changes locally before submitting the PR (e.g. some of the markdown parsing is not always reliable with regard to line-breaks etc.).
• There is a strong dependency between the website and the documentation. Separating them could mean the website would need to know which rollup version to check out to retrieve the documentation at some point. Or that we are limited in how we can advance the documentation.
• There also is a strong dependency between the website REPL and rollup to similar effects.
• If we want to be able to access older versions of the documentation, we could let the website directly access Github and retrieve the corresponding markdown files from the right version. If the repo that is checked out is the rollup repo or a separate website repo should not make any difference.

@guybedford Let me know your thoughts about this. As I will be on vacation starting Wednesday without a computer, I will most likely not be able to set anything up before end of next week.

[guybedford]

I'm definitely agreed on merging the website content into a docs folder here.

If this was a static site, I'd say we could have the website static generator code as a dependency of Rollup that builds into the output folder for the website fine. But since the site needs to run as a server and can't do static generation I really wouldn't want it to be integrated into the main repo.

[lukastaegert]

I'm definitely agreed on merging the website content into a docs folder here.

Then let's start with this. Though this would mean that for the automatic release, rollup would

• check out the website repo in a local folder
• copy the documentation into that folder as well
• trigger the release
• delete everything again

Which seems a little clumsy. And it would not have any of the advantages I listed above. But it would work, and the documentation is the single most important thing I want.

[guybedford]

copy the documentation into that folder as well

Using Rollup as a dependency of the website, the documentation and Rollup version can be read from require.resolve('rollup/package.json') and require.resolve('rollup/docs').

trigger the release

Since the site runs as a server, it should be possible to have a script in the site itself that plugs into the GitHub API and checks for releases periodically, updating the Rollup dependency of itself, and then rebuilding the repl and docs.

delete everything again

Nothing wrong with checking in the built assets for the website itself if necessary?

[lukastaegert]

Since the site runs as a server, it should be possible to have a script in the site itself that plugs into the GitHub API and checks for releases periodically, updating the Rollup dependency of itself, and then rebuilding the repl and docs

As the site is built locally before being released, I do not think it is easily possible to tell the server to rebuild itself but I may be wrong. Also having the server polling for updates sounds wasteful and requires a lot more work while my my proposal would be trivial to set up and not require any polling.

Nothing wrong with checking in the built assets for the website itself if necessary?

We could also add a script to rollup that automatically commits and merges the new documentation into the website once a new version is released (by the CI) which should trigger the existing documentation release procedure.

[eddiemonge]

Please don't merge all the plugins into a monorepo. It makes finding issues specific to a plugin so very difficult.