Repo Documentation + Wiki as Submodule

[igorbrigadir]

There's the docs on https://twarc-project.readthedocs.io/en/latest/twarc2/ which are generated from https://github.com/DocNow/twarc/tree/v2/docs

Separately, we have https://github.com/DocNow/twarc/wiki

In wonder if it's worth trying to unify those? A Wiki in a repo is itself a git repository, so we can add the wiki as a submodule like this:

https://coderwall.com/p/bwpwwg/the-wiki-as-submodule-pattern

This might be worth it at the cost of a couple of extra git commands. Not sure how well that would work yet because i haven't tried it but it looks like a good idea.

[edsu]

I would be open to this. But we could also treat the wiki as a place where documents are developed and then moved into the docs folder when they are ready? I like the idea of making it more automatic though!

[igorbrigadir]

Yeah i'm gonna try this setup on a different repo to see how it would work, i got the idea but i haven't actually tried it.

The advantage is that things are always in one place, and people can contribute on github, the disadvantage is maybe a couple more git commands when developing / building docs etc.

[edsu]

@igorbrigadir I added you as a maintainer on ReadTheDocs so feel free to try things out there. If anyone else wants access please send me your ReadTheDocs username and I will add you.

[edsu]

I don't know if you've made much progress on this @igorbrigadir but I'm starting to feel like it might be useful to use the wiki as a proving ground for docs which, if appropriate, can be moved into the docs folder when they are ready. Then the wiki could be updated to say that the page has moved? I think this could allow some documentation to grow organically and then we can cherry pick things that work as part of the official documentation? What do you think?

[igorbrigadir]

the wiki as a proving ground for docs which, if appropriate, can be moved into the docs folder when they are ready.

Yes! With this submodule way it should be possible to have this workflow. Just depends on what order you update / push / pull the submodules.

I'll merge them together first and make some example changes to make sure i got the right idea.

[igorbrigadir]

Had to make some small changes to enable submodules on readthedocs and in Github Actions for this to work.

Might need to write a wiki doc with instructions on how to edit it but now it works like this:

The existing docs are exactly the same as before. https://twarc-project.readthedocs.io/en/latest/

I added wiki Pages section as an example of pulling in pages from the wiki into the main docs. Any edits to the wiki will work as is - we can put drafts in like you'd expect. The barrier to entry is allowing contributions in the repo settings. You can also manually edit the files on your machine and commit them directly.

Readthedocs will build all pages but it won't display them all by default, so we can show only the complete ones. We can continue to edit the docs as markdown files just like earlier, except with a couple of extra commands:

When developing, an extra command needs to be run for the first time only:

git submodule update --init

After this, docs appears just as another folder.

If you cd into docs it behaves as a separate git repo, for example:

Main repo:

twarc$ git status
On branch v2
Your branch is up-to-date with 'origin/v2'.

Submodule:

twarc/docs$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

If making changes to the docs, you would have to commit the changes to the submodule, or edit the wiki on github, and then update the main repo too, eg:

cd docs
*make some changes*
git commit ...
git push ...

Now on the main repo:

twarc$ git status
On branch v2
Your branch is up-to-date with 'origin/v2'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   docs (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

Now git push from main repo will build the latest docs.

Alternatively, a change from the github web interface:

*Make some changes on Wiki *

cd to to docs folder and pull latest changes

twarc/docs$ git pull
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 3), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), done.
From https://github.com/DocNow/twarc.wiki
   7539e2e..c5cf6f7  master     -> origin/master
Updating 7539e2e..c5cf6f7
Fast-forward
 wiki/Home.md | 2 ++
 1 file changed, 2 insertions(+)

Again, from the main repo, commit and push:

twarc$ git commit -m "merge docs changes from wiki"
[v2 fc2241a] merge docs changes from wiki
 1 file changed, 1 insertion(+), 1 deletion(-)

git push will update the docs on https://twarc-project.readthedocs.io/en/latest/ etc.

Let me know what do you make of this setup.

[edsu]

This is great @igorbrigadir. I may move a few things around in the side menu. I guess we could make tie ReadTheDocs to the docnow/twarc.wiki.git repository instead of docnow/twarc? That way any push of changes to that repository will trigger a rebuild of the documentation? What do you think?

[igorbrigadir]

I guess we could make tie ReadTheDocs to the docnow/twarc.wiki.git repository instead of docnow/twarc?

I don't know for sure but i think this might break the click generated docs from the source files.

while i think giving readthedocs the wiki repo link directly could also work, if you want to do this, there may need to be some config changes for that in https://github.com/DocNow/twarc/blob/v2/.readthedocs.yaml since it's not a submodule anymore - and https://github.com/DocNow/twarc/blob/v2/mkdocs.yml would also have to move into the wiki repo, and we might loose the ability to auto generate the docs like https://github.com/DocNow/twarc/wiki/twarc2 that gets turned into https://twarc-project.readthedocs.io/en/latest/twarc2/

[edsu]

Ah yes, we would lose the ability to generate the API docs, unless the submodule relation were inverted. Lets leave it the way it is for now. It feels a bit complicated to me, but I suspect I'll get used to it quickly.

[igorbrigadir]

Based on this GitHub actions thing I think I could add an action that would auto update everything on readthedocs on demand or on every change
https://github.community/t/how-to-updade-repo-wiki-from-github-actions/121151/5 which would make it work for everything, auto generated from code and from wiki

[edsu]

I'm not sure I want to introduce GitHub actions for documentation just yet. I'm confused why ReadTheDocs does not appear to be pulling from what's on master in DocNow/twarc.wiki.git -- do you see what's going on @igorbrigadir?

[edsu]

Oh i see from the GitHub interface that the docs folder is pegged to a particular revision:

https://github.com/DocNow/twarc

I'm not entirely sure how that happened, maybe I did it accidentally somehow?

[igorbrigadir]

Yeah - that's just how the submodule generally works - it stays with the commit until you explicitly update it in the main repo.

This is awkward, because you have to commit and push twice.. once in the sub-module, and once in the main repo.

But, i just found a good way to make it sync both ways, with #426 any edits to the wiki will be auto updated in the main repo, so that all the generated docs can be built too, and published to readthedocs.

This way the workflow is:

** Make changes to wiki **
https://github.com/DocNow/twarc/wiki

Docs get commited to wiki & main repo, and updated on readthedocs.

That should be fairly simple to work with, the entire interface is via Github Wiki edits in markdown.

Or alternatively, for changes where the github wiki interface isn't enough, or major changes are needed:

** clone repo, if it's the first time run git submodule update --init, go into docs folder, make changes **

push changes to docs first:

cd docs
git commit ...
git push 

then push change to main:

cd ..
git add docs
git commit -m "update docs"
git push

[edsu]

This feels a bit too confusing to me. I'm going to try to simplify things by having the docs generated directly from twarc.wiki -- if it doesn't seem right I can flip it back with the v2.0.0 tag.

[edsu]

Unfortunately readthedocs doesn't seem to get push notifications from twarc.wiki. I also ran into some trouble with the import of twarc.version in setup.py

I'm concerned that this wiki/readthedocs setup is a bit too clever so going to flip things back to having the docs directly in twarc and we can move things from the wiki when we decide to.

[edsu]

I apologize to roll back this experiment @igorbrigadir but I would like to make it easier to reason about what and where the documentation is, and how it is built. Having it here in the twarc module will help with that I think.

[igorbrigadir]

No problem at all!

It got a bit too complicated indeed, and the extra overhead is too much effort for what it's worth.

There are still a few leftover parts like with: submodules: recursive in https://github.com/DocNow/twarc/blob/main/.github/workflows/main.yml that extra part won't break anything if it stays or goes.

Separately, i pulled out #428 because it's useful without submodules.