Migrate help.openstreetmap.org from OSQA

[simonpoole]

ASKBOT is an actively maintained fork of qsqa (that was last worked on something like 5 years ago), see:
https://github.com/ASKBOT/askbot-devel and https://askbot.com/

If we want to continue to run our own Q&A site migrating would make sense and would likely address many of the issues we have with QSQA (not just that it is abandonware). The alternative would be to give up running our own system and move to stackexchange, which has IMHO some downsides including loosing the content I believe.

Other issues to consider:

• while ASKBOT still works with mysql, they support postgres better, I assume that is OK with us in any case?
• there is, for unfathomable reasons, no off the shelf migration script/facility, this implies that some coding is likely to be involved. Potentially however the people as ASKBOT would be willing to do this for a reasonably small some of money and that would help everybody using QSQA..

Comments @woodpeck @systemed and anybody else answering questions on help?

[tomhughes]

If it's a fork I'm surprised the database can't just be migrated or have their been schema changes on the OSQA side since the fork?

[simonpoole]

The later probably not, and I wouldn't rule out that we could migrate the mysql DB and then migrate to Postgres. From reading the support forums it is not quite clear what the issue is, but I'll dig a bit more.

This http://askbot.org/doc/upgrade.html may be part of the problem.

[EdLoach]

Sounds sensible to move to something that is actively maintained if it isn't too difficult to do.

[woodpeck]

OSQA works well enough for me, the glitches that are there don't bother me. I have never used Askbot, it doesn't exactly look like there's a huge developer community behind it apart from the company who runs it - will anything remain if their business tanks? I don't mind staying with OSQA but I don't mind switching either. If that should mean losing our content I think that wouldn't even be too bad, many the answers in OSQA are outdated. We count pull a read-only static image from OSQA for reference purposes and copy over relevant stuff when a question pops up... plus a reboot would create a fair karma playing field where you don't have this woodpeck guy with twice as many karma points as the next contender.

[tomhughes]

@woodpeck problem is that I need to upgrade that machine to 16.04 at some point but I have no confidence OSQA won't break (it did when we went to 14.04 and we had to patch it).

[SomeoneElseOSM]

If it makes sense to move (and I can see why there would, given the issues we had and to some extent still have), I don't see a particular problem with doing so.

There are some "old answers" that are worth preserving, but bluntly some sort of static HTML would work for those. Many more old answers "were correct ages ago" but now no longer are.

I don't think that a "flattening" of user reputation would be a problem either.

[SomeoneElseOSM]

I've been asked to add a link to https://wiki.openstreetmap.org/wiki/Talk:Help.openstreetmap.org#Migration_QSQA_-.3E_ASKBOT here, but the content of the message is small enough to post in full:

"Sounds quite good for me. However, if we are migrating to anywhere we rather should think about how to solve the issue of outdated answers. Just deleting the whole thing every 7 years is a waste. There ARE many still good answers (yes, also much irrelevant or even wrong answers nowadays) and I try to take care of them (updating, organizing) as good as possible. I also would like to have incoming link not to be broken if possible (yes, setting it up as static html clone might work). Regarding the "outdated" problem, also see the section above. Regarding possible Karma flattening: that is likely no problem. --Aseerel4c26 (talk) 20:04, 21 February 2017 (UTC)"

[scaidermern]

Keeping good questions/answers is certainly a good idea. We should make sure people are still able to find them via search engines.

[woodpeck]

Assuming for a moment that copying over part but not all of existing Qs&As is possible. perhaps we could share the load of judging which questions/answers to keep and which to kill by resetting all scores to 0, and then asking everyone to look through stuff and upvote what they think should be kept...

[scaidermern]

Yes. Maybe we can go through the top 50 questions here and decide which ones to keep: https://help.openstreetmap.org/questions/?sort=mostvoted

If they can't get migrated maybe we could still copy'n'pasting the question and 1-2 good answers and then mark them as community wiki.

[pnorman]

I think moving off of qsqa is the only reasonable option given it's unmaintained nature and problems that have required patching, so it's only a question of if askbot the best option.

There is information on migrating, so it's just a question of how far back the migrations go.

[grischard]

The Wireshark project solved this by simply archiving their osmqa site.

[tomhughes]

The problem is that I need to upgrade the machine and doing so will almost certainly break OSQA meaning that even running it in RO mode is likely to be hard.

[simonpoole]

@tomhughes @gravitystorm @zerebubuth this would seem like a fairly self-contained activity that could be farmed out to an interested person from the wider community that wants to help. Again objections to raising this on the a suitable mailing list (I'm pushing a bit because besides the discussed issues, we should be fixing the gravatar support in some way from a data protection pov, and it doesn't make sense to do this in the old unsupported code).

[tomhughes]

If you can find somebody that can figure out how to migrate the data then sure.

I actually have a chef branch where I started work on code to install and manage askbot.

[RobJN]

Is this any help?
https://gist.github.com/cholin/171898332d63afb8371c

[nebulon42]

How big is the MySQL database? Can one get access to a dump?

[tomhughes]

It's postgres, not mysql. Ignoring the (vast) session table it looks like there is a little over 100Mb in the tables.

If you're volunteering to attempt a conversion then we could probably come to an arrangement to make it available, subject to appropriate agreements around safeguarding any private data.

[nebulon42]

I know I shouldn't because I have too many projects already, but somehow this interests me. :)
So let's say I volunteer to attempt a conversion without any commitment to an ETA. But without trying it's all theory and if not successful there might be at least lessons learned for the next attempt.
I'm fine with setting up a NDA or something similar. Being part of CWG might help already though there are no written agreements of course.

[slachiewicz]

Maybe we can try with Discourse - more active development also Postgress and Ruby based.
https://github.com/discourse/discourse/blob/master/docs/INSTALL.md

[ppKrauss]

Hi, about @slachiewicz's suggestion,

Maybe we can try with Discourse ...

See meta.discourse/porting-from-askbot-to-discourse ... Seems that the major investment for us is to enhance this embrionary adaptation.

[tomhughes]

Discourse is a forum (or a replacement for forums) not a Q+A site.

That is kind of evident in the fact that it comes from one of the founders of Stack Overflow and was specifically designed to address a different use case...

[slachiewicz]

As you can see @ppKrauss in the repository, you can boast a large and active number of migration scripts https://github.com/discourse/discourse/tree/master/script/import_scripts

[simonpoole]

@ppKrauss @slachiewicz pls no. While discourse may have its merits as a forum replacement (just that I've never found any) it is really completely at odds with what the help site is supposed to be doing (as @tomhughes has said).

[ppKrauss]

Some news about our options: meta.discourse community's positions and suggestions.

[pnorman]

So, I had a look at AskBot looking at testing a conversion.

I can't recommend it because we'll have the same problems as now.

All versions of AskBot rely on Django versions that were released 3-4 years ago and are EOL. The commit activity on its repository is minimal.

Keeping OSQA on a machine running 16.04 14.04! is not a long-term option. Running Django 1.8 which has stopped getting security patches is also not a long-term solution.

[slachiewicz]

As workaround maybe pack all Django OSQA stuff to Docker?

[pnorman]

As workaround maybe pack all Django OSQA stuff to Docker?

That would still be running 14.04 for everything except the kernel, so there'd be the same problems.

Since 14.04 hits EOL in April, we need to upgrade the machine by then, don't we?

Should we set sometime in March as a deadline for we're going to upgrade the machine, even if it causes problems with help.osm.org?

[pnorman]

I looked at the options listed on a SE question and these are what I see as an active alternative

• question2answer code
• scoold code (Might not be fully open source)

Other options that are not open-source are

• AnswerHub, by the people who wrote OSQA.
• Stack Exchange Enterprise
• Try to get a Stack Exchange created for OSM
• Drop all OSMF support for a Q&A site. Stack Exchange might fill the gap, or might not

Nothing stands out as the right option that is better than all alternatives.

[simonpoole]

We could naturally pay the askbot developer to migrate askbot to a current django version (this would naturally only solve one of many issues).

[lonvia]

Instead of using a full Docker container, the current OSQA can be simply run in a Python virtualenv with the older python libraries. Doesn't solve any of the major issue either but at least gives an upgrade path for the machine.

[rory]

Another reason to switch from OSQA: It doesn't support all OSM usernames. I cannot log into help.openstreetmap.org with my OSM account (rorym 🏳️‍🌈).

[tomhughes]

Could we please keep this vaguely on the subject @rory - that is nothing to do with OSQA in any case and everything to do with our custom authentication plugin and my non-existent python skills.

[EdLoach]

Looking at pnorman's comment and in particular question2answer I did find this migration script from OSQA in case it proves to be useful, or at least a start.

[ppKrauss]

Discourse seems the only active and secure for a long-term investment.

Please, who likes the idea of migrating to "Discourse + Q&A" (there are two customizable options, Solved and Voting) click here "like"...

If 3 or more people like it, we can study how to adapt and migrate, then, after study, come back here with a solution proposal.

---

PS: Discourse can be used also as commom software infrastructure for other chanels, like the forum (see example in other multi-community).

[thomersch]

I briefly talked with the Askbot developer earlier this year, it looks like there is some activity. On the 0.11 branch they have a Django 2.2 compatible version, which is compatible with Python 3.5/3.6/3.7. They also have a Dockerfile nowadays which could help with isolating the installation.

[simonpoole]

@thomersch IMHO it is quite possible that the OSMF could pay the dev for help in/doing the migration and any missing bits that might need to be developed. Maybe you could start a conservation with them/him about this?

[thomersch]

@simonpoole Sure, will do

[pnorman]

@simonpoole Sure, will do

@thomersch did the conversation with the askbot developer happen?

[PeterMortensen]

There is the upcoming Codidact - codidact.org (open source splinter from Stack Exchange).

The meta site for it is on Discourse, but it is not based on Discourse.

[grischard]

Askbot development seems to have slowed down significantly.

[mwoehlke-kitware]

BTW, I was pointed over here from this question. I'd like to point out that the current site is just plain broken by third-party cookie blocking, which is a fairly minimal thing for anyone even remotely privacy-conscious to enable, and can't be easily disabled on a per-site basis. (This is because it relies on Google for jQuery, and Google won't serve that unless they can set their own cookies. Privacy-conscious folks should be screaming in horror about now...) The only thing I can do on the site is ask questions. I can't comment, can't vote, and suspect I can't even answer, and previews and tags are completely non-functional.

Despite comments here to the contrary, the current OSQA instance cannot be considered functional by reasonable definitions.

[Kovoschiz]

BTW, I was pointed over here from this question. I'd like to point out that the current site is just plain broken by third-party cookie blocking, which is a fairly minimal thing for anyone even remotely privacy-conscious to enable, and can't be easily disabled on a per-site basis. (This is because it relies on Google for jQuery, and Google won't serve that unless they can set their own cookies. Privacy-conscious folks should be screaming in horror about now...) The only thing I can do on the site is ask questions. I can't comment, can't vote, and suspect I can't even answer, and previews and tags are completely non-functional.

Despite comments here to the contrary, the current OSQA instance cannot be considered functional by reasonable definitions.

Fyi, tomhughes/osqa@577d530 appears to fix the thing mostly, as commented in that question later. Answering, reporting, and editing (forgot about favoriting) worked fine before. See http://gis.19327.n8.nabble.com/our-Q-A-site-help-openstreetmap-org-is-dying-tp5966020p5969974.html as well.

[cellio]

Hello. Codidact team member here. We might be able to help you if you're still interested in setting up your community elsewhere. We have a hosted network of communities and you're welcome to propose yours, or if you're looking for self-hosting, our project is open-source but nobody else has set it up yet, so we don't have a nice package with good documentation that you can just unpack and go. https://codidact.org/ has links for our Discord server (for chat), our GitHub repo, and our existing communities including our network-wide meta. Please feel free to get in touch if you want to explore options.

If this is already settled and no longer of interest, I apologize for the intrusion.

[GoodClover]

Just mentioning that from the Codidact Discord there was link for these scripts as a basis for seeing how migrating data may be possible.

They are for importing Stack Exchange data, written in Ruby so I don't know much about how they work.

[ArtOfCode-]

Just further from @cellio's comment - more than happy to provide assistance with installing a self-hosted instance of Codidact if that's what y'all would be looking for. If it's easier to have a community on our existing network, we can help with content import and custom domains, etc, as necessary.

[natrius]

What are the features that are expected by you guys from a QA? Because some things can be done with discourse. I'm just interested in what is expected because it is also part of the discussion #377 there.