Migrate wiki.sagemath.org contents to more suitable places

[mkoeppe]

The problem with wiki.sagemath.org:

It excludes new and newish developers who do not have the required legacy Trac account but log in to Trac using their GitHub accounts.

Wikis work because of the principle of crowdsourcing. Without a crowd, the material will just become outdated. New material added is indistinguishable from old material.

There is also a conflict between viewing the Wiki (1) as an archive (as a record of past achievements, and perhaps for the benefit of future software archaeologists); (2) as something that is useful for current users and developers; and (3) a place for discussion and planning actions.

These uses are not compatible. We propose to separate these types of material clearly and move each to a more appropriate system.

Source material on wiki.sagemath.org:

• https://wiki.sagemath.org/SageWikiMaintenance%20page
• https://wiki.sagemath.org/TitleIndex

---

Proposed targets:

Migration to GitHub issues

Ticket/issue descriptions can be used for collecting/curating/staging material that eventually should be added to our documentation or to our code.

• Meta-ticket: document configuring IDEs and text editors #30500 Meta-ticket: document configuring IDEs and text editors
• Meta-ticket: Add downstream (external user) packages as optional/experimental packages #31164 Meta-ticket: Add user packages from https://wiki.sagemath.org/SageMathExternalPackages as optional/experimental packages

Migration to our documentation

(needs conversion to rst)

• Developer's Guide: Update reviewer's check list from wiki #33649 Developer's Guide: Update reviewer's check list from wiki
• Developer guide: Check material in https://wiki.sagemath.org/TentativeConventions #33663 Developer guide: Check material in https://wiki.sagemath.org/TentativeConventions
• Merge https://wiki.sagemath.org/CodeSharingWorkflow into developer manual #34554 Merge https://wiki.sagemath.org/CodeSharingWorkflow into developer manual

Current drawback: The documentation is only updated one every stable release (to be fixed in #33862)

either: Migration to www.sagemath.org

(needs conversion to html; or website scripts need extension to handle markdown or moinmoin sources; https://python-markdown.github.io/reference/; https://github.com/github/markup)

• This would be suitable for example for archiving https://wiki.sagemath.org/Workshops

• Migrate old Sage release tours and changelogs from Sage website and wiki to GitHub Releases or Sage documentation #31533 Migrate old Sage release tours from Sage wiki

• Merge info from https://wiki.sagemath.org/CategoryHomepage into contributors.xml

or: Migration to GitHub wiki:

This is where we moved the contents of the Trac wiki.

"Note: Search engines will not index the contents of wikis. To have your content indexed by search engines, you can use GitHub Pages in a public repository." https://docs.github.com/en/communities/documenting-your-project-with-wikis/about-wikis

---

Considered but won't be needed:

Migration to GitHub pages:

https://pages.github.com/

(needs conversion to markdown)

Unclear why we should create a new website when we already have www.sagemath.org

---

Format conversion tools, workflow:

• https://github.com/swenson/sagewiki -> forked as https://github.com/sagemath/sagewiki-to-github

• Convert directly to Markdown using https://github.com/phlash/moin2markdown

• https://pypi.org/project/moin2gitwiki/

• To do while migrating: We also audit/update our documentation regarding links to the wiki.

  $ git grep 'wiki[.]sage'
  

References:

• http://moinmo.in/HelpOnConfiguration

CC: @slel @dimpase @sagetrac-tmonteil @fchapoton @saraedum @kwankyu @williamstein @haraldschilly @jhpalmieri

Component: website/wiki

Issue created by migration from https://trac.sagemath.org/ticket/33725

[sagetrac-tmonteil]

comment:4

Please don't. In terms of infrastructure, the wiki is pretty well hosted, while the VM that hosts the trac+wiki is unmaintained, so this looks like a regression to me (at least right now).

[dimpase]

comment:5

I think that trac is better, as it allows contributions with github logins. And the bus factor of the present wiki hosting is rather small.

[mkoeppe]

comment:6

It's critical that we get rid of the old Sage wiki. As Dima says - it excludes new developers.

Trac is maintained because it has to be (thanks to all who keep it running!) The wiki part of it places no additional burden on maintaining Trac.

[mkoeppe]

Branch: u/mkoeppe/migrate_wiki_sagemath_org_to_trac_sagemath_org_wiki

[sagetrac-git]

Commit: 57fc849

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

57fc849

configure.ac: Remove outdated advice regarding exotic platforms

[mkoeppe]

Dependencies: #33088

[sagetrac-git]

Changed commit from 57fc849 to 45c1a39

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

b46c953	README: Document --enable-editable and --enable-download-from-upstream-url
a3ffe84	README.md: Instructions for Apple Silicon
d7708df	README.md: Restructure macOS prep instructions
2e1d22d	src/doc/en/installation/source.rst: Remove instructions for Solaris, AIX, HP-UX
9ca39a2	README.md: Point to developer guide for git details
f1e1166	src/doc/en/installation/source.rst: Point to README.md for step-by-step instructions
3f6c6e0	src/doc/en/developer/workspace.rst: Link to Trac #30500
8716d80	src/doc/en/installation/source.rst: Update required venv Python extension modules from build/pkgs/python3/spkg-configure.m4
10c9b87	Merge #33088
45c1a39	src/doc/en/developer/trac.rst: Update wiki URL for patchbot

[mkoeppe]

comment:17

Some pages, for example https://wiki.sagemath.org/SageMathExternalPackages, are written in RST.

The Trac wiki also supports RST - https://trac.sagemath.org/wiki/WikiRestructuredText - but it looks like it is not activated.

@dimpase, could you install docutils (as per https://trac.sagemath.org/wiki/WikiRestructuredText#Requirements)?

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

da69787

src/doc: Fix 'hardcoded link could be replaced'

[mkoeppe]

comment:60

I went through the same steps of thinking.
The only thing that's missing is support for .md files in our website scripts.

[kwankyu]

comment:61

Replying to Matthias Köppe:

The only thing that's missing is support for .md files in our website scripts.

That is an important point. Together with liberal write permission to developers, that will make a search-engine-indexed website close to a wiki site...

[dimpase]

comment:62

one can hopefully have the content versioned in git (on GitHub), and then updating automatically, via GitHub Actions, is doable (it's not instant WYSIWYG, but OK)

[mkoeppe]

comment:63

Yes, the source code of our website is maintained on github. https://github.com/sagemath/website

I agree, it would be good to add GH Actions that runs what Harald currently runs manually after merging PRs, with a preview of the generated HTML on PRs.

[kwankyu]

comment:66

If we end up running self-hosted gitlab (for mirroring, archiving, or replacing github) instead of trac, then we can move wiki to the gitlab as it also has wiki system.

[kwankyu]

comment:71

After trac-to-github migration, we will have a new wiki https://github.com/sagemath/sage/wiki. The new wiki will have good visibility for sage developers. Developers could collaboratively edit the wiki pages easily.

Its weak point "search engines will not index the contents of github wikis" seems minor. The wiki is for developers, who know already where to search. For prospective new users, we can advertise the new wiki through the sage homepage.

We don't need to close wiki.sagemath.org. It is being used and loved by some developers. We can just have two wikis. Why not? They will compete for a while, but I guess the new wiki will eventually win the game (the old wiki will be deprived of contents and visitors in the long run).

[mkoeppe]

comment:72

Replying to Kwankyu Lee:

After trac-to-github migration, we will have a new wiki https://github.com/sagemath/sage/wiki. [...]
Its weak point "search engines will not index the contents of github wikis" seems minor.

I think I can agree with this viewpoint.

If there is material that needs to be found by search engines, we can just migrate that material to either the documentation or the website.

[mkoeppe]

comment:73

Replying to Kwankyu Lee:

We don't need to close wiki.sagemath.org. It is being used and loved by some developers. We can just have two wikis. Why not? [...]

That's also fine with me, at least for a transition period. I still think that we should soon migrate away from the old wiki.sagemath.org, but that should not be a precondition for the migration away from Trac.

[mkoeppe]

I've removed the material from the description that became outdated with our completed migration of Trac, including the Trac wiki, to GitHub.

[kwankyu]

You didn't update the title?

[mkoeppe]

Thanks

[mkoeppe]

I have run a test migration using @swenson's script, now forked at https://github.com/sagemath/sagewiki-to-github

The input was the archive dump.wiki.2022-10-10.tar.gz provided by @sagetrac-tmonteil.
The output is at https://github.com/sagemath/sagewiki_to_gh - note that this is a big repository, which preserves the version history of all pages.

I have only looked at a few files (the release tours). The conversion is not perfect, but not bad either.
The script complained about a few things, see log.

convert-2023-02-12.log

[kwankyu]

I agree that the conversion is rather good.

I find that sage 9.5 release tour is missing.

[kwankyu]

The input was the archive dump.wiki.2022-10-10.tar.gz provided by @sagetrac-tmonteil.

Getting the latest data is the first thing to do.

The output is at https://github.com/sagemath/sagewiki_to_gh - note that this is a big repository, which preserves the version history of all pages.

What do you mean? I don't see version history.

[mkoeppe]

https://github.com/sagemath/sagewiki_to_gh/commits/master

[mkoeppe]

sage 9.5 release tour is missing.

Yes, the log has several messages about this (and other pages).

[kwankyu]

After editing the new wiki for a while, I am not sure about the value of moving around old wiki pages only of historical value. By moving them to a new place, we only break links to them and often break the pages themselves.

In that sense, old wiki pages that we never plan to update would better to remain (perhaps freezed) at its current place.

This applies to trac wiki pages that we already converted and moved, but also to sage wiki pages that we didn't move yet.

If we keep trac wiki running forever as an archive, we may remove old pages from the new wiki. (then we shouldn't have rewritten links in issues to wikis, but we already did and this is irreversible.)

What would be the rationale of moving old sage wiki pages to our new wiki? (this is the conflict described in the issue description)

[kwankyu]

https://github.com/sagemath/sagewiki_to_gh/commits/master

I see. That is nice.