Skip to content
This repository has been archived by the owner on Nov 9, 2023. It is now read-only.

should the qiime forum move to BioStars? #1926

Open
gregcaporaso opened this issue Feb 4, 2015 · 58 comments
Open

should the qiime forum move to BioStars? #1926

gregcaporaso opened this issue Feb 4, 2015 · 58 comments

Comments

@gregcaporaso
Copy link
Contributor

Starting at the Biocore 14.10 summit, we've been discussing the future of the QIIME Forum, and whether we should migrate from Google Groups. There are a few core motivations for this:

  • The Google Groups searching functionality is surprisingly bad, making it hard for users to find posts related to their questions.
  • Because there is no up-voting/down-voting functionality (as is present e.g. on StackOverflow) it is very hard to find the "right answer" in long discussion threads. The threads are effectively chains of email discussion, which is very inconvenient.
  • Because it's hard to see or show-off individual contributions, there is less incentive for users to answer each others questions. If users had some page that they could point people to that highlighted their contributions (again, like StackOverflow does, for example), this could be linked from their websites, LinkedIn profiles, etc to "show off" their QIIME knowledge and support contributions.

We discussed three options for doing this. The first was just pointing users to StackOverflow to ask their questions there. This isn't really the intended use of StackOverflow, and the maintainers frown upon that. The next was to use Biology StackExchange, but that is more for general biology questions, not bioinformatics technical support. Finally, we discussed using BioStars (web, paper), which is more directly intended for this purpose.

I've been discussing using BioStars with the site admin and senior author on the paper, @ialbert. He prefers that we don't just direct our users there as that could overwhelm BioStars since we have such a large community. He suggested two options by email - I'm paraphrasing here:

  1. That we run our own BioStars (this is done for Galaxy and BioConductor), and could probably be done with an AWS instance fairly easily.
  2. That we (quoting @ialbert) "wait for Biostar 3.0 (I plan to release that by March 1st) - one of the goals is to turn it more into like Reddit - where there are different communities (groups) and one has the option of subscribing to communities instead of using tags. For example groups would be using subdomains where linking to say: qiime.biostars.org". @ialbert, is there a prototype of this anywhere right now that we could start looking at?

I think both of these sound like good possibilities, and it sounds like we may even be able to export previous discussions from the Google Group to initially populate the forum (is that accurate @ialbert?). With respect to option 2, I would think that instead of using qiime.biostars.org, we would start by using qiime1.biostars.org, which would give us a natural way to separate out support for QIIME 2, which will be very different. @ialbert said that this would work fine.

Opening this up for developer and community feedback, there are a few specific questions that would be good to get input on.

  • Does using BioStars (either option 1 or 2) seem like a good move forward for the QIIME Forum (i.e., do you think it would help us get around the specific issues I noted above)?
  • Which of those options (1 or 2) do you prefer, and why?
  • What specific features would you like to see in the next iteration of the QIIME Forum (regardless of whether we were to use BioStars or not)?
  • Any other thoughts on this?

Thanks for the input!

@antgonza
Copy link
Contributor

antgonza commented Feb 4, 2015

👍 to BioStar option 2. Mainly because we could brand the forums as
you point out but also because we will not need to do any sys admin.

Features:

  • Improved search engine. To be honest, almost anything will be better
    that how Google groups search currently works.
  • It will be great that upon submission the system, shows you a list
    of similar questions/answers and ask you if you are sure if any of
    those do not solve your question.
  • Import google groups questions/answers. I will not mind spending a
    day or two merging and checking good questions/answers to the forum,
    if the extracting/merging tool was user friendly.
  • Keep track of not only helping users but also questions asked + a
    way to rank the question based on difficulty. This can help us
    identify power users so we encourage them to become qiime masters.

@ElDeveloper
Copy link
Member

Thanks for starting this discussion. My opinion is geared towards moving
away from google groups as a support forum, as there are several other
options that provide more useful functionality for this purpose.

I'm all for using Biostars option 2, as @antgonza points out that way we
wouldn't have to do any system administration, but we would still have
control over the site.

One of the features that I would like to make sure we have is email
updates i. e. get an email everytime someone posts a new question or a
new answer to an existing question. It sounds masochistic, but it's a
good way to keep track of problems users are having.

It will also be nice and fun to test this out internally before we
actually launch it.

On (Feb-04-15| 9:12), Greg Caporaso wrote:

Starting at the Biocore 14.10 summit, we've been discussing the future of the QIIME Forum, and whether we should migrate from Google Groups. There are a few core motivations for this:

  • The Google Groups searching functionality is surprisingly bad, making it hard for users to find posts related to their questions.
  • Because there is no up-voting/down-voting functionality (as is present e.g. on StackOverflow) it is very hard to find the "right answer" in long discussion threads. The threads are effectively chains of email discussion, which is very inconvenient.
  • Because it's hard to see or show-off individual contributions, there is less incentive for users to answer each others questions. If users had some page that they could point people to that highlighted their contributions (again, like StackOverflow does, for example), this could be linked from their websites, LinkedIn profiles, etc to "show off" their QIIME knowledge and support contributions.

We discussed three options for doing this. The first was just pointing users to StackOverflow to ask their questions there. This isn't really the intended use of StackOverflow, and the maintainers frown upon that. The next was to use Biology StackExchange, but that is more for general biology questions, not bioinformatics technical support. Finally, we discussed using BioStars (web, paper), which is more directly intended for this purpose.

I've been discussing using BioStars with the site admin and senior author on the paper, @ialbert. He prefers that we don't just direct our users there as that could overwhelm BioStars since we have such a large community. He suggested two options by email - I'm paraphrasing here:

  1. That we run our own BioStars (this is done for Galaxy and BioConductor), and could probably be done with an AWS instance fairly easily.
  2. That we (quoting @ialbert) "wait for Biostar 3.0 (I plan to release that by March 1st) - one of the goals is to turn it more into like Reddit - where there are different communities (groups) and one has the option of subscribing to communities instead of using tags. For example groups would be using subdomains where linking to say: qiime.biostars.org". @ialbert, is there a prototype of this anywhere right now that we could start looking at?

I think both of these sound like good possibilities, and it sounds like we may even be able to export previous discussions from the Google Group to initially populate the forum (is that accurate @ialbert?). With respect to option 2, I would think that instead of using qiime.biostars.org, we would start by using qiime1.biostars.org, which would give us a natural way to separate out support for QIIME 2, which will be very different. @ialbert said that this would work fine.

Opening this up for developer and community feedback, there are a few specific questions that would be good to get input on.

  • Does using BioStars (either option 1 or 2) seem like a good move forward for the QIIME Forum (i.e., do you think it would help us get around the specific issues I noted above)?
  • Which of those options (1 or 2) do you prefer, and why?
  • What specific features would you like to see in the next iteration of the QIIME Forum (regardless of whether we were to use BioStars or not)?
  • Any other thoughts on this?

Thanks for the input!


Reply to this email directly or view it on GitHub:
#1926

@gregcaporaso gregcaporaso changed the title should the qiime forum move to Biostars? should the qiime forum move to BioStars? Feb 4, 2015
@ialbert
Copy link

ialbert commented Feb 4, 2015

Let me first refer to the following as a partial brain dump of new features for Biostars 3.0: ialbert/biostar-central#291

  • a prototype should be ready in a few weeks
  • email integration already exists (subscribe/reply to posts) but needs to be improved
  • exporting your list into an mbox or some simple, easy to parse format would help populating the site and make past content available (you may need to use an automated crawler though, I saw posts on this). We've migrated 10 years of bioconductor from emails into threaded posts.

@rob-knight
Copy link

Option 2 sounds good to me as March is so close and we won't have bandwidth before then anyway due to AGBT deadline for qiita...

@gregcaporaso
Copy link
Contributor Author

Thanks for the link @ialbert, the feature list looks really relevant to what we want to be able to do.

I also agree that option 2 with BioStars sound like a good way to go, and that March 1 would work fine.

@cleme
Copy link
Contributor

cleme commented Feb 5, 2015

Thanks for bringing this up @gregcaporaso. I agree that Google Groups is less than ideal, and particularly so for search (not necessarily the only reason why similar questions are posted multiple times in the Qiime forum, though...)

I'm also with @ElDeveloper in the need for email updates and testing before switching. If I am reading it correctly biostars has 15,000 users while the qiime forum has over 2000, so we would add more than 10% new users and would be good to know that won't be an issue.

As for other alternatives, StackExchange had a proposal for a bioinformatics group that was closed for lack of traction, and the biology one is too broad (although there is a "bioinformatics" label). Seqanswers might be worth discussing though, not that I advocate for it but would like to know what others think

@ialbert
Copy link

ialbert commented Feb 5, 2015

@cleme on scaling I don't expect problems until an order of magnitude change. Mostly because even with tens of thousands of users the number of simultaneous writes is not that large. Page reads are much easier to optimize. The site maintained the same performance while traffic doubled last year(without actually having to do anything) CPUs are still only at about 17% average load and we can easily provision more compute power if needed.

@marina-yurieva
Copy link

Option 2 sounds better. I'm already on BioStars and it is more convenient to have everything in one place instead of checking many different forums.

@alk224
Copy link

alk224 commented Feb 6, 2015

Will the existing cache of forum discussion be able to be migrated to biostars? I have referenced some posts there and it looks like a good system, but I would hate for all the existing shared knowledge to be left behind in the google group.

And now I just reread @ialbert's comment. Looks like this could happen.

@ialbert
Copy link

ialbert commented Feb 6, 2015

I think it is important to migrate as much content as possible to maintain
an continuity and have search engines pick up the new location

While Google does not seem to offer an export capability, searches indicate
that people have published solutions of tools that are set up and crawl it.
If you can produce any type of organized data I can write you an import
script.

(as I mentioned before we have worked closely with the Bioconductor team
and helped them migrate all their content- 10 or so years worth, that too
took some workarounds as some posts were not in the mbox file but were
referenced via urls on another site).

On 6 February 2015 at 16:44, alk224 notifications@github.com wrote:

Will the existing cache of forum discussion be able to be migrated to
biostars? I have referenced some posts there and it looks like a good
system, but I would hate for all the existing shared knowledge to be left
behind in the google group.


Reply to this email directly or view it on GitHub
#1926 (comment).

@RNAer
Copy link

RNAer commented Feb 10, 2015

all for option 2. Compared to a standalone qiime biostar, another advantage of it is to allow us to easily connect and interact with broader and diverser scientific communities that is steadily increasing in Biostar, which is clearly beneficial for both qiime users and developers.

@colinbrislawn
Copy link
Contributor

👍 for option 2, waiting for Biostar 3.0 and creating a qiime community there.

The upvote/downvote system would greatly highlight the best solutions and facilitate relevant discussions.

A separate community for qiime users would help us answer the boring (but important!) questions about qiime without disrupting the diverse questions of the 'general' Biostar.

@ialbert It's good to see you on a github page! I look forward to using Biostar 3.0

@gregcaporaso
Copy link
Contributor Author

@ialbert, do you guys have an update on BioStar 3? I poked around the website and github repo and didn't see anything. We'd love to test out a prototype whenever one is ready.

@ialbert
Copy link

ialbert commented Mar 6, 2015

no there is massive working being done, it is very close to a functional demo - perhaps one week off

https://github.com/ialbert/biostar-central/commits/3.0

off schedule mainly because we had way too many school closing days due to the weather

@gregcaporaso
Copy link
Contributor Author

Ok, thanks, good luck!

@ialbert
Copy link

ialbert commented Mar 13, 2015

Progress has been good. I expect the new Biostar demo site to be operational next week. What would make a big difference if your explored how the QIIME google group content could be exported into a text format. Some sort of web scraping might be needed as Google Groups does not seem to offer downloads. But there's got to be software/solutions to do that.

Any type of reasonably defined output format would work - I'll write the importer into Biostar.

Frankly in the light of Google turning off Google Code I think they will do the same to Google Groups. I want to make Biostar fully compatible with importing Google Groups.

@ebolyen
Copy link
Member

ebolyen commented Mar 13, 2015

That's awesome @ialbert! I can work on figuring out ways to serialize a google group. Will report back with any solutions here. A brief google seems to indicate they offer no API support whatsoever (what a surprise!), so it probably will require a web-scraper.

@ialbert
Copy link

ialbert commented Mar 13, 2015

Here is an idea! If someone has the emails in their email client that could be exported/saved into mbox. I could use that.

@ebolyen
Copy link
Member

ebolyen commented Mar 13, 2015

That would definitely be easy, but I don't know how many people would have a complete record. On the note of scraping, I am having a lot of success so far with PhantomJS and selenium. Everything is rendering and I am able to find elements. I think a robust solution would be to have the user put in the first few topics that are in the forum and the scraper can figure out the rest (it would seem google's dom classes are relatively dynamic and change all of the time).

@colinbrislawn
Copy link
Contributor

@ialbert I've got about two years of old emails from the qiime google group saved to my gmail. @ me if you need my help.

@ialbert
Copy link

ialbert commented Mar 13, 2015

@colinbrislawn if you can save it as an mbox email it to me istvan.albert at gmail.com and I take a loot. We can also compare that to what @ebolyen can scrape off

@colinbrislawn
Copy link
Contributor

@ialbert I would need to find a way to filter out the rest of my email from the mbox file. I'll take at look at downloading and parsing it.

Would this still be worthwhile if I only have a fraction (say 70%) of the qiime forum?

@ialbert
Copy link

ialbert commented Mar 13, 2015

@colinbrislawn I am sure there are tools to filter mbox files like http://pamoller.com/mboxfilter.html

even a subset of emails helps, the site needs to be seeded so that new users find the new site rather than the old

@adamrp
Copy link
Contributor

adamrp commented Mar 13, 2015

Seems like it might be worth waiting for a general solution from @ebolyen
rather than have to write a separate (and ultimately redundant) importer
for 2 yrs of emails, although it us a good thought and an impressive back
catalog from @colinbrislawn :)
On Mar 13, 2015 3:47 PM, "Istvan Albert" notifications@github.com wrote:

@colinbrislawn https://github.com/colinbrislawn I am sure there are
tools to filter mbox files like http://pamoller.com/mboxfilter.html

even a subset of emails helps, the site needs to be seeded so that new
users find the new site rather than the old


Reply to this email directly or view it on GitHub
#1926 (comment).

@colinbrislawn
Copy link
Contributor

Google lets you elegantly sort and export your archived emails, so I now have my old qiime-forum emails in an .mbox file, if anyone wants it.

However, the google group only sends the first part of each message. Like this:

=============================================================================
Topic: Error converting tab delimited OTU file to BIOM format
Url: http://groups.google.com/group/qiime-forum/t/2f766aff4a1d6f24
=============================================================================

---------- 1 of 3 ----------
From: Dylan Lennon <djlenno@gmail.com>
Date: Jan 25 09:38AM -0800
Url: http://groups.google.com/group/qiime-forum/msg/d02fa9c32579ef41

#OTUID  P410.AMicroDM1.S13.1190 P410.AMicroJL1.S13.1214 P410.AMicroDL3.S13.1212 P410.AMicroIL1.S13.1220 P410.AMicroJM1.S13.$

---------- 2 of 3 ----------
From: zhenjiang zech xu <zhenjiang.xu@gmail.com>
Date: Jan 25 11:23AM -0700
Url: http://groups.google.com/group/qiime-forum/msg/227867e52a948882

what manipulation did you do? and can you be specific on " the taxonomy
summary plots were gibberish"? maybe post a snapshot



---------- 3 of 3 ----------
From: Dylan Lennon <djlenno@gmail.com>
Date: Jan 25 01:07PM -0800
Url: http://groups.google.com/group/qiime-forum/msg/a67b14165198ded8

Hi Zech,

I attached the .txt files (for L2-L6) which, from what I understand,
contain the raw data used to generate the plots. If you open them, you can
see that the taxonomy is, for example

It would be easy to match that opening text block and the following conversations, but I'm not sure this if useful if all the messages are truncated.
@ialbert, Would you still like to see the .mbox file or should we forget this and wait for @ebolyen?

Edit: I was receiving short emails because I had set my Google Groups notification to 'Abridged email.' If someone was using 'All email' this process would work perfectly.

@colinbrislawn
Copy link
Contributor

When I was exporting my Google emails, I discovered you can elegantly export other Google products. Google Forums was on the list, but I was not able to export anything because I don't run this forum.

Perhaps the admin of qiime-forum@googlegroups.com could try to export the forum?
Link to export google data: https://www.google.com/settings/takeout

@colinbrislawn
Copy link
Contributor

@ElDeveloper It's too bad that the export does not work.

@ialbert I was receiving short emails because I had my Google Groups notification set to 'Abridged email.' If someone was using 'All email,' their emails would have the full messages.

@gregcaporaso
Copy link
Contributor Author

@ElDeveloper, I can make you an admin on the Qiime Forum if that will help
you with the export. Let me know.

On Mon, Mar 16, 2015 at 12:37 PM, Colin Brislawn notifications@github.com
wrote:

@ElDeveloper https://github.com/ElDeveloper It's too bad that the
export does not work.

@ialbert https://github.com/ialbert I was receiving short emails
because I had my Google Groups notification set to 'Abridged email.' If
someone was using 'All email,' their emails would have the full messages.

Reply to this email directly or view it on GitHub
#1926 (comment).

@ElDeveloper
Copy link
Member

I'm an admin, but that didn't work regardless.

On (Mar-22-15|15:19), Greg Caporaso wrote:

@ElDeveloper, I can make you an admin on the Qiime Forum if that will help
you with the export. Let me know.

On Mon, Mar 16, 2015 at 12:37 PM, Colin Brislawn notifications@github.com
wrote:

@ElDeveloper https://github.com/ElDeveloper It's too bad that the
export does not work.

@ialbert https://github.com/ialbert I was receiving short emails
because I had my Google Groups notification set to 'Abridged email.' If
someone was using 'All email,' their emails would have the full messages.

Reply to this email directly or view it on GitHub
#1926 (comment).


Reply to this email directly or view it on GitHub:
#1926 (comment)

@ialbert
Copy link

ialbert commented Mar 23, 2015

This week a beta test will commence here is the QIIME group:

http://qiime.test.biostars.org/

Some functionality may be broken or not performing as intended or as I am
still hooking things up. The site could also be periodically offline etc.

But overall it shows the main feature of isolation of posts across groups
and building independent communities.

Main site:

http://test.biostars.org/

vs QIIME specific site:

http://qiime.test.biostars.org/

On 22 March 2015 at 18:20, Yoshiki Vázquez Baeza notifications@github.com
wrote:

I'm an admin, but that didn't work regardless.

On (Mar-22-15|15:19), Greg Caporaso wrote:

@ElDeveloper, I can make you an admin on the Qiime Forum if that will
help
you with the export. Let me know.

On Mon, Mar 16, 2015 at 12:37 PM, Colin Brislawn <
notifications@github.com>
wrote:

@ElDeveloper https://github.com/ElDeveloper It's too bad that the
export does not work.

@ialbert https://github.com/ialbert I was receiving short emails
because I had my Google Groups notification set to 'Abridged email.' If
someone was using 'All email,' their emails would have the full
messages.

Reply to this email directly or view it on GitHub
#1926 (comment).


Reply to this email directly or view it on GitHub:
#1926 (comment)


Reply to this email directly or view it on GitHub
#1926 (comment).

@colinbrislawn
Copy link
Contributor

@ialbert I'm glad this is up and running, even if it's not ready for prime time.

I've changed my email settings so that I receive full copies of qiime-forum posts. This means that I have a running copy of the whole qiime-forum from March 2015 onward. You mentioned the importance of 'seeding' biostars with important older posts so that users will google their way into biostars, and I completely agree.

If we shift to biostars in summer, I will be able to export at least four months of qiime-forum posts.

@ElDeveloper
Copy link
Member

@ialbert, the site looks very neat, congrats on the awesome work! I just
signed up, I would like to see what the email support is like, so I'm
looking forward to someone creating some posts and get the discussion
started.

Thanks @colinbrislawn!

On (Mar-23-15| 9:51), Colin Brislawn wrote:

@ialbert I'm glad this is up and running, even if it's not ready for prime time.

I've changed my email settings so that I receive full copies of qiime-forum posts. This means that I have a running copy of the whole qiime-forum from March 2015 onward. You mentioned the importance of 'seeding' biostars with important older posts so that users will google their way into biostars, and I completely agree.

If we shift to biostars in summer, I will be able to export at least four months of qiime-forum posts.


Reply to this email directly or view it on GitHub:
#1926 (comment)

@gregcaporaso
Copy link
Contributor Author

Hi all, The qiime-specific site seems to be redirecting me to the main site
(I go to http://qiime.test.biostars.org/ and end up at
http://test.biostars.org/). Is that expected?

On Mon, Mar 23, 2015 at 10:36 AM, Yoshiki Vázquez Baeza <
notifications@github.com> wrote:

@ialbert, the site looks very neat, congrats on the awesome work! I just
signed up, I would like to see what the email support is like, so I'm
looking forward to someone creating some posts and get the discussion
started.

Thanks @colinbrislawn!

On (Mar-23-15| 9:51), Colin Brislawn wrote:

@ialbert I'm glad this is up and running, even if it's not ready for
prime time.

I've changed my email settings so that I receive full copies of
qiime-forum posts. This means that I have a running copy of the whole
qiime-forum from March 2015 onward. You mentioned the importance of
'seeding' biostars with important older posts so that users will google
their way into biostars, and I completely agree.

If we shift to biostars in summer, I will be able to export at least
four months of qiime-forum posts.


Reply to this email directly or view it on GitHub:
#1926 (comment)

Reply to this email directly or view it on GitHub
#1926 (comment).

@ialbert
Copy link

ialbert commented Mar 26, 2015

just go ahead and create a new group called qiime , this is a test site that periodically gets reset as I test stuff.

today I pushed out a new version but looks like I forgot to recreate the group

@ialbert
Copy link

ialbert commented Mar 26, 2015

You can create a group here

http://test.biostars.org/g/list/

@gregcaporaso
Copy link
Contributor Author

Thanks, just did that here:
http://qiime1.test.biostars.org/

On Thu, Mar 26, 2015 at 3:08 PM, Istvan Albert notifications@github.com
wrote:

You can create a group here

http://test.biostars.org/g/list/

Reply to this email directly or view it on GitHub
#1926 (comment).

@gregcaporaso
Copy link
Contributor Author

This looks really nice @ialbert.

Have we had any progress on figuring out how to scrape all of the content from Google Groups? cc @ElDeveloper @ebolyen @colinbrislawn I don't have all of the Qiime Forum emails sent to me directly (I get the digest), but maybe someone else does which would let us try the mbox idea, @antgonza maybe?

@ebolyen
Copy link
Member

ebolyen commented Mar 27, 2015

So I have a very primitive scraper, unfortunately, it is really a one-time thing, as it depends on so many specifics of the UI. Additionally google group's pagination doesn't actually go much past 30, so I have to manually scroll the index on phantomJS (which means I need to figure out how to clear the dom once in a while or it will take a stupid amount of memory). All in all, it seems possible, but disgusting. I can spend a bit more time on this if you would like, but I don't have high hopes for a general solution. Another problem is the actual pages returned have slightly different layouts on occasion, so I may get to a point 2000+ messages in, and suddenly whatever I had assumed about the DOM no longer holds.

@ebolyen
Copy link
Member

ebolyen commented Mar 27, 2015

@gregcaporaso also, while there is a lot of content on the google groups, I strongly suspect we don't want most of it. Almost all of doesn't match the format of a Q/A and it would largely drown out anything else on the site. I think it may be more work to effectively seed from the entire history, then even trying to get at it in the first place.

@ialbert
Copy link

ialbert commented Mar 27, 2015

If you can get 2000 messages out that would a pretty good start.

the way emails are transformed into the Q&A format is that the thread starter becomes "the question", replies to the thread starter become answers and replies to answers become comments.

So one thread from the email list becomes just one page on the site. Also all quoted text is dropped since the replies are also on the page.

@ebolyen
Copy link
Member

ebolyen commented Mar 27, 2015

I'm not confident that their DOM structure is that expressive (such that I could determine the reply relationships) but I will look into that as a step. There's basically 3 unsolved issues in that case:

  • How not to run out of memory. (I would hope I could just mutate the DOM, but I fear there will be ghost elements from the javascript)
  • How to retrieve the useful parts of a thread and their relationships (I can dump rendered HTML at this point, but I haven't attempted much beyond this... yet)
  • How to deal with the random failures of the google group's client itself (hangs sometimes and needs a reload for no reason) and how to deal with inconsistent DOM (one particular thread is paginated, I'll need to figure out how to recognize that and visit each page)

@ialbert
Copy link

ialbert commented Mar 27, 2015

here is an idea: put out a call on the user group - some people may have the full emails that they could export

@gregcaporaso
Copy link
Contributor Author

@ialbert, we'll chat about this today and get back to you on it. Thanks!

@gregcaporaso
Copy link
Contributor Author

@antgonza found a few possible approaches to scraping google groups:

http://saturnboy.com/2010/03/scraping-google-groups/
blanchardjeremy/google-groups-php-api#1 this is a feature request on a project that hasn't had a commit since 2011
https://www.google.com/settings/takeout this just gives you the list of users

@jairideout
Copy link
Member

I agree with @ebolyen that seeding Biostars with the QIIME forum google group may not be that effective.

Based on @ialbert's description of how the import would work, I think we'll end up with a question per thread (which makes sense), but the first reply to the thread starter will become the one (and only) answer, and all following messages will be comments on the "answer". Often times, this "answer" will be a developer asking for more details, etc., so it isn't really an answer. I don't think we'll ever see a question with more than one "answer"; it will always be the first reply to the thread starter. Also, the answer will be polluted with a huge number of comments because these threads often contain a large number of messages.

I'm worried that if we populate Biostars this way, we won't be setting a good precedent for how QIIME users should use Biostars to get help. Since we're fundamentally changing how we provide support, is it feasible to keep the current QIIME forum around in a read-only format and treat it as an archive? Some selective seeding of Biostars would help too.

@alk224
Copy link

alk224 commented Mar 27, 2015

Second that, Jai. As a user, I don't want to see that old stuff. Plus so
much of the forum is old enough as to be outdated from questions about
scripts that are no longer used or have changed.

Andrew Krohn
PhD Candidate, NAU Biological Sciences
Research Technician, EnGGen

On Fri, Mar 27, 2015 at 8:24 AM, Jai Ram Rideout notifications@github.com
wrote:

I agree with @ebolyen https://github.com/ebolyen that seeding Biostars
with the QIIME forum google group may not be that effective.

Based on @ialbert https://github.com/ialbert's description of how the
import would work, I think we'll end up with a question per thread (which
makes sense), but the first reply to the thread starter will become the one
(and only) answer, and all following messages will be comments on the
"answer". Often times, this "answer" will be a developer asking for more
details, etc., so it isn't really an answer. I don't think we'll ever see a
question with more than one "answer"; it will always be the first reply to
the thread starter. Also, the answer will be polluted with a huge number of
comments because these threads often contain a large number of messages.

I'm worried that if we populate Biostars this way, we won't be setting a
good precedent for how QIIME users should use Biostars to get help. Since
we're fundamentally changing how we provide support, is it feasible to keep
the current QIIME forum around in a read-only format and treat it as an
archive? Some selective seeding of Biostars would help too.


Reply to this email directly or view it on GitHub
#1926 (comment).

@rob-knight
Copy link

Perhaps this is too radical a suggestion but what if we limp along with the current forum until qiime 2.0 and make a clean break then?

On Mar 27, 2015, at 8:26 AM, alk224 <notifications@github.commailto:notifications@github.com> wrote:

Second that, Jai. As a user, I don't want to see that old stuff. Plus so
much of the forum is old enough as to be outdated from questions about
scripts that are no longer used or have changed.

Andrew Krohn
PhD Candidate, NAU Biological Sciences
Research Technician, EnGGen

On Fri, Mar 27, 2015 at 8:24 AM, Jai Ram Rideout <notifications@github.commailto:notifications@github.com>
wrote:

I agree with @ebolyen https://github.com/ebolyen that seeding Biostars
with the QIIME forum google group may not be that effective.

Based on @ialbert https://github.com/ialbert's description of how the
import would work, I think we'll end up with a question per thread (which
makes sense), but the first reply to the thread starter will become the one
(and only) answer, and all following messages will be comments on the
"answer". Often times, this "answer" will be a developer asking for more
details, etc., so it isn't really an answer. I don't think we'll ever see a
question with more than one "answer"; it will always be the first reply to
the thread starter. Also, the answer will be polluted with a huge number of
comments because these threads often contain a large number of messages.

I'm worried that if we populate Biostars this way, we won't be setting a
good precedent for how QIIME users should use Biostars to get help. Since
we're fundamentally changing how we provide support, is it feasible to keep
the current QIIME forum around in a read-only format and treat it as an
archive? Some selective seeding of Biostars would help too.


Reply to this email directly or view it on GitHub
#1926 (comment).


Reply to this email directly or view it on GitHubhttps://github.com//issues/1926#issuecomment-86974151.

@cleme
Copy link
Contributor

cleme commented Mar 27, 2015

I favor Rob's idea. Seeding the old posts into Biostars does not seem trivial, so we could keep things under google forums for now when previous posts can still be useful for reference. Once we transition to Q2 I think those older posts will probably be less relevant

@colinbrislawn
Copy link
Contributor

I support @rob-knight's idea as well. A clean start matches the QIIME 2.0.0 design philosophy.

Edit: The more I think about this, the more I like it.
This will ONLY be populated by issues related to the new qiime. No seeding == no outdated issues.
As people start to use the qiime 2.0.0 release candidate, they can ask questions on biostars, letting us test both, then go live concurrently.

@jairideout
Copy link
Member

👍

@gregcaporaso
Copy link
Contributor Author

Ok, I agree that this probably is the way to go (i.e., wait until QIIME 2).

@gregcaporaso
Copy link
Contributor Author

And in the meantime, we can start using the new BioStars for other projects, such as scikit-bio, biom, Qiita(?), etc?

@rob-knight
Copy link

Sounds good…

On Mar 27, 2015, at 11:46 AM, Greg Caporaso <notifications@github.commailto:notifications@github.com> wrote:

And in the meantime, we can start using the new BioStars for other projects, such as scikit-bio, biom, Qiita(?), etc?


Reply to this email directly or view it on GitHubhttps://github.com//issues/1926#issuecomment-87049859.

@gregcaporaso
Copy link
Contributor Author

Thinking more about this...

Since we get support questions for many of our different tools on the QIIME Forum, maybe we should create a general biocore forum on BioStars, and start directing the non-QIIME questions there. We can tag questions based on software package, and if any QIIME stuff does come up there we can tag as QIIME1 (ultimately vs QIIME 2).

Google Groups is just so bad. Since it's so hard to find answers, we are just answering the same questions over and over. And we do need forums for skbio (SO probably isn't the most appropriate for this as we've learned by investigating it as an option for the QIIME Forum), Qiita, BIOM, etc.

So in this case we wouldn't actively transition the QIIME Forum now, but start directing new non-QIIME questions to BioStars. We'd then colocate all of the biocore forums in one place using tags to differentiate, which ultimately would reduce effort because it's less to monitor, easier to link questions or tag as related to multiple projects, and it's not just dense threads of email discussions where it's impossible to tell correct answers from garbage (i.e., Google Groups).

@antgonza
Copy link
Contributor

antgonza commented Apr 17, 2015 via email

@gregcaporaso
Copy link
Contributor Author

Quick follow up on this. I got an updated ETA from @ialbert on BioStars 3, and it should be available later in the summer. He said that it's ready for release, but he's waiting until he gets back from vacation in August to release. I think that timing is fine for us.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests