RFC: split this repository in five different (maybe more) repositories

[JJ]

The problem

The main problem is testing time #2498. But that's not the only one. There are so many things in this repo, with more added every month, that it comes a time when it's become unmanageable, with author tests only run from time to time due to time restrictions, for instance, and the requisites of some tools conflicting with the requisites from others. The different parts are also too tightly coupled (described somewhere as a "rats nest") that it's difficult to think about testing (see #2529 #2539) lots of parts (including the main site-generation app) that are not being tested right now.
The five parts here are:

• The documents themselves. They are housed in the docs/ directory. Some auxiliary files are in writing-docs and of course t/ and xt/ directories.
• p6doc. This is an user-facing script which should be used to consult/search documentation, but which has right now lots of issues: p6doc - 'make html' fails #2014 "p6doc -f exit" doesn't work #2532 Fix p6doc build errors #2103 p6doc IO::Path on windows #2534. This uses a set of files, but it's not clear which ones; Build.pm6 in the root dir is one of them.
• The website docs.perl6.org It's difficult to find out where p6doc ends and this starts. Mainly htmlify.p6 (which has lots of issues: Update htmlify.p6 to fix 'got empty filename' issue #1971.  #1997 Refactor htmlify.p6 #1823 Eliminate disambiguation from htmlify.p6 #2499 )
• Auxiliary modules used by these above, including Pod::Documentable (see Spin off Perl6::Documentable #1937 ) and many others under the lib/ directory.
• Other different files that have some relationship with this repo, including the Dockerfile, utility files that might or might not be part of the first block

Suggestions

I already proposed #2295 to general indifference, but I think we should go further tnan that, leaving the docs in this directory, and filter commits to create all new repositories that keep the history, which could be called App::doc, doc-site, several repos, one for each lib Module which is functionally independent and doc-kerfile (pun intended).

[moritz]

Isn't that approach a bit backwards? Shouldn't you/we first reduce coupling, and then possibly move things into a separate repositories, should it be still necessary afterwards?

If we just split without reducing coupling, we're in for a world of pain without any benefits.

[lizmat]

I agree that there should first be an effort to reduce coupling. But such an effort only makes sense if we decide on splitting into multiple repos as a final goal. What would otherwise be the incentive for reducing coupling?

[moritz]

@lizmat you can achieve easier testing, and run tests for decoupled components, even if you don't split into repos.

(FTR I don't really care about the splitting either way, as long as it doesn't happen before the decoupling).

[JJ]

@moritz the problem is that most of the stuff outside the documentation is not tested. In order to reduce coupling (or rat-nesting) we would need to devise a good amount of tests; but in order to to that, we need to eliminate coupling since it's not clear what's there because it needs to be there or what's simply a side effect. It's really a catch-22 situation, so I would really start by splitting, which will force decoupling.
Mind you, we would not tear down the site meanwhile; we'll approach that piecewise and in the best fashion possible.
@lizmat reducing coupling is needed because it's impossible right now to fix some of the bugs listed above unless we refactor the whole thing, and refactoring needs testing, and testing needs decoupling. Also, I don' t really think there should be a single documentation repo to serve them all, and also to download it all! People who want to use p6doc shouldn't need to install Mojolicious (which is a dependency of the site, not the app).

[Altai-man]

FWIW, one can get a fresh repo clone and cut it in pieces locally. Any number of file moving or changes can be done without breaking anything in current process.

When one will achieve a separate set of components(directories), instructions for each one and other things like that ready, it will be easy to just git init & commit & push given directoried into created repositories.
I am not sure it is a piece of work that necessarily has to be shared for various contributors to be able to be involved. Even if it's like that, a throwaway repo can be created with all partially separated directories included.

[finanalyst]

I am actively working on this whole set of problems.
I agree with JJ. There are several sets of inter-related issues.

1. The documentation of the language.
2. A web interface to the documentation, including the possibility to edit the documentation.
3. A local interface to the documentation, such as p6docs.
4. Support software, such as Pod::To::HTML.
   I do not think there should be an immediate split because there is too much coupling.
   My goal is to replicate the website without any coupling.
   Then get a way for p6docs to work separately.

[pmurias]

Splitting the repository contents into a bunch of subdirectories that are still in the repository itself is an option.
This would maintain a degree of separation while still maintaining the ability to work on multiple things together without juggling PRs/commits across multiple repositories.
node.js land example: https://github.com/parcel-bundler/parcel/tree/master/packages/core

[coke]

A related discussion just came up on #2662 - Specifically addressing the thought of splitting out some items from the documentation itself - if the goal is to have targeted distributions of docs via p6doc, or a copy of the website, that is something that can be done by generating user-specific artifacts at regular interviews (say, quarterly) for the users. IMO, this isn't something we should solve at the git repo level. That said, having it installable (say via a CPAN module via zef) would be great, but we don't need separate repos to accomplish that.

Additionally, one of the comments on #2662 was regarding the size of downloads for the user; note that removing a file from HEAD doesn't mean it's not in the repo anymore; it's still there and being transferred, so any solution that simply removes the file from one repo and puts a copy in the other isn't helping on that front, if they're using git. (And if they're not using git, that's another pro for same repo, different artifacts, IMO)

Regarding pulling out modules - it very much depends on purpose of those modules; Pod::Cache, for example, is very basic and only for build/testing, and exists as a module only to avoid code duplication. (The issue that it's untested is a separate one and can be addressed in place.) Pulling it out into a separate repo complicates development for little benefit.

[JJ]

The thing is that we should create different repositories for different targets. Right now this repo includes everything and the kitchen sink: tools for doccers, the documentation itself, tools for generating an HTML file from the documentation, the p6doc... That makes difficult to refactor, makes testing take a loooong time, makes difficult to attract new contributors... Splitting would have all kinds of benefits for developers, with little or no negative impact on user experience.

[coke]

Strongly disagree that breaking the repository into smaller repositories is the correct approach.

[JJ]

@lizmat this is the "big split" issue.

[coke]

Strongly disagree that breaking the repository into smaller repositories is the correct approach.

Several of these steps were already done, please feel free to ignore my old opinion here for the remaining items.

[coke]

@JJ , can you summarize what's left to do before we can close out this ticket?

[JJ]

Will try to next week. El sáb, 16 abr 2022 a las 18:40, Will Coleda ***@***.***>) escribió:

…

@JJ <https://github.com/JJ> , can you summarize what's left to do before we can close out this ticket? — Reply to this email directly, view it on GitHub <#2542 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAD5FHMKIJLVKMNNHVRILVFLUO5ANCNFSM4GONS3WQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- JJ

[coke]

p6doc has been migrated out; the site generation will be included with @Altai-man's new site, Documentable, etc.

I think the main thrust of this ticket has been accomplished. Once we complete the spin off of the web site, we'll revisit and open tickets for remaining/new issues as we find them.

Thanks, everyone.