-
Notifications
You must be signed in to change notification settings - Fork 268
should the qiime forum move to BioStars? #1926
Comments
👍 to BioStar option 2. Mainly because we could brand the forums as Features:
|
Thanks for starting this discussion. My opinion is geared towards moving I'm all for using Biostars option 2, as @antgonza points out that way we One of the features that I would like to make sure we have is email It will also be nice and fun to test this out internally before we On (Feb-04-15| 9:12), Greg Caporaso wrote:
|
Let me first refer to the following as a partial brain dump of new features for Biostars 3.0: ialbert/biostar-central#291
|
Option 2 sounds good to me as March is so close and we won't have bandwidth before then anyway due to AGBT deadline for qiita... |
Thanks for the link @ialbert, the feature list looks really relevant to what we want to be able to do. I also agree that option 2 with BioStars sound like a good way to go, and that March 1 would work fine. |
Thanks for bringing this up @gregcaporaso. I agree that Google Groups is less than ideal, and particularly so for search (not necessarily the only reason why similar questions are posted multiple times in the Qiime forum, though...) I'm also with @ElDeveloper in the need for email updates and testing before switching. If I am reading it correctly biostars has 15,000 users while the qiime forum has over 2000, so we would add more than 10% new users and would be good to know that won't be an issue. As for other alternatives, StackExchange had a proposal for a bioinformatics group that was closed for lack of traction, and the biology one is too broad (although there is a "bioinformatics" label). Seqanswers might be worth discussing though, not that I advocate for it but would like to know what others think |
@cleme on scaling I don't expect problems until an order of magnitude change. Mostly because even with tens of thousands of users the number of simultaneous writes is not that large. Page reads are much easier to optimize. The site maintained the same performance while traffic doubled last year(without actually having to do anything) CPUs are still only at about 17% average load and we can easily provision more compute power if needed. |
Option 2 sounds better. I'm already on BioStars and it is more convenient to have everything in one place instead of checking many different forums. |
Will the existing cache of forum discussion be able to be migrated to biostars? I have referenced some posts there and it looks like a good system, but I would hate for all the existing shared knowledge to be left behind in the google group. And now I just reread @ialbert's comment. Looks like this could happen. |
I think it is important to migrate as much content as possible to maintain While Google does not seem to offer an export capability, searches indicate (as I mentioned before we have worked closely with the Bioconductor team On 6 February 2015 at 16:44, alk224 notifications@github.com wrote:
|
all for option 2. Compared to a standalone qiime biostar, another advantage of it is to allow us to easily connect and interact with broader and diverser scientific communities that is steadily increasing in Biostar, which is clearly beneficial for both qiime users and developers. |
👍 for option 2, waiting for Biostar 3.0 and creating a qiime community there. The upvote/downvote system would greatly highlight the best solutions and facilitate relevant discussions. A separate community for qiime users would help us answer the boring (but important!) questions about qiime without disrupting the diverse questions of the 'general' Biostar. @ialbert It's good to see you on a github page! I look forward to using Biostar 3.0 |
@ialbert, do you guys have an update on BioStar 3? I poked around the website and github repo and didn't see anything. We'd love to test out a prototype whenever one is ready. |
no there is massive working being done, it is very close to a functional demo - perhaps one week off https://github.com/ialbert/biostar-central/commits/3.0 off schedule mainly because we had way too many school closing days due to the weather |
Ok, thanks, good luck! |
Progress has been good. I expect the new Biostar demo site to be operational next week. What would make a big difference if your explored how the QIIME google group content could be exported into a text format. Some sort of web scraping might be needed as Google Groups does not seem to offer downloads. But there's got to be software/solutions to do that. Any type of reasonably defined output format would work - I'll write the importer into Biostar. Frankly in the light of Google turning off Google Code I think they will do the same to Google Groups. I want to make Biostar fully compatible with importing Google Groups. |
That's awesome @ialbert! I can work on figuring out ways to serialize a google group. Will report back with any solutions here. A brief google seems to indicate they offer no API support whatsoever (what a surprise!), so it probably will require a web-scraper. |
Here is an idea! If someone has the emails in their email client that could be exported/saved into mbox. I could use that. |
That would definitely be easy, but I don't know how many people would have a complete record. On the note of scraping, I am having a lot of success so far with PhantomJS and selenium. Everything is rendering and I am able to find elements. I think a robust solution would be to have the user put in the first few topics that are in the forum and the scraper can figure out the rest (it would seem google's dom classes are relatively dynamic and change all of the time). |
@ialbert I've got about two years of old emails from the qiime google group saved to my gmail. @ me if you need my help. |
@colinbrislawn if you can save it as an mbox email it to me istvan.albert at gmail.com and I take a loot. We can also compare that to what @ebolyen can scrape off |
@ialbert I would need to find a way to filter out the rest of my email from the mbox file. I'll take at look at downloading and parsing it. Would this still be worthwhile if I only have a fraction (say 70%) of the qiime forum? |
@colinbrislawn I am sure there are tools to filter mbox files like http://pamoller.com/mboxfilter.html even a subset of emails helps, the site needs to be seeded so that new users find the new site rather than the old |
Seems like it might be worth waiting for a general solution from @ebolyen
|
Google lets you elegantly sort and export your archived emails, so I now have my old qiime-forum emails in an .mbox file, if anyone wants it. However, the google group only sends the first part of each message. Like this:
It would be easy to match that opening text block and the following conversations, but I'm not sure this if useful if all the messages are truncated. Edit: I was receiving short emails because I had set my Google Groups notification to 'Abridged email.' If someone was using 'All email' this process would work perfectly. |
When I was exporting my Google emails, I discovered you can elegantly export other Google products. Google Forums was on the list, but I was not able to export anything because I don't run this forum. Perhaps the admin of qiime-forum@googlegroups.com could try to export the forum? |
@ElDeveloper It's too bad that the export does not work. @ialbert I was receiving short emails because I had my Google Groups notification set to 'Abridged email.' If someone was using 'All email,' their emails would have the full messages. |
@ElDeveloper, I can make you an admin on the Qiime Forum if that will help On Mon, Mar 16, 2015 at 12:37 PM, Colin Brislawn notifications@github.com
|
I'm an admin, but that didn't work regardless. On (Mar-22-15|15:19), Greg Caporaso wrote:
|
This week a beta test will commence here is the QIIME group: http://qiime.test.biostars.org/ Some functionality may be broken or not performing as intended or as I am But overall it shows the main feature of isolation of posts across groups Main site: vs QIIME specific site: http://qiime.test.biostars.org/ On 22 March 2015 at 18:20, Yoshiki Vázquez Baeza notifications@github.com
|
@ialbert I'm glad this is up and running, even if it's not ready for prime time. I've changed my email settings so that I receive full copies of qiime-forum posts. This means that I have a running copy of the whole qiime-forum from March 2015 onward. You mentioned the importance of 'seeding' biostars with important older posts so that users will google their way into biostars, and I completely agree. If we shift to biostars in summer, I will be able to export at least four months of qiime-forum posts. |
@ialbert, the site looks very neat, congrats on the awesome work! I just Thanks @colinbrislawn! On (Mar-23-15| 9:51), Colin Brislawn wrote:
|
Hi all, The qiime-specific site seems to be redirecting me to the main site On Mon, Mar 23, 2015 at 10:36 AM, Yoshiki Vázquez Baeza <
|
just go ahead and create a new group called qiime , this is a test site that periodically gets reset as I test stuff. today I pushed out a new version but looks like I forgot to recreate the group |
You can create a group here |
Thanks, just did that here: On Thu, Mar 26, 2015 at 3:08 PM, Istvan Albert notifications@github.com
|
This looks really nice @ialbert. Have we had any progress on figuring out how to scrape all of the content from Google Groups? cc @ElDeveloper @ebolyen @colinbrislawn I don't have all of the Qiime Forum emails sent to me directly (I get the digest), but maybe someone else does which would let us try the mbox idea, @antgonza maybe? |
So I have a very primitive scraper, unfortunately, it is really a one-time thing, as it depends on so many specifics of the UI. Additionally google group's pagination doesn't actually go much past 30, so I have to manually scroll the index on phantomJS (which means I need to figure out how to clear the dom once in a while or it will take a stupid amount of memory). All in all, it seems possible, but disgusting. I can spend a bit more time on this if you would like, but I don't have high hopes for a general solution. Another problem is the actual pages returned have slightly different layouts on occasion, so I may get to a point 2000+ messages in, and suddenly whatever I had assumed about the DOM no longer holds. |
@gregcaporaso also, while there is a lot of content on the google groups, I strongly suspect we don't want most of it. Almost all of doesn't match the format of a Q/A and it would largely drown out anything else on the site. I think it may be more work to effectively seed from the entire history, then even trying to get at it in the first place. |
If you can get 2000 messages out that would a pretty good start. the way emails are transformed into the Q&A format is that the thread starter becomes "the question", replies to the thread starter become answers and replies to answers become comments. So one thread from the email list becomes just one page on the site. Also all quoted text is dropped since the replies are also on the page. |
I'm not confident that their DOM structure is that expressive (such that I could determine the reply relationships) but I will look into that as a step. There's basically 3 unsolved issues in that case:
|
here is an idea: put out a call on the user group - some people may have the full emails that they could export |
@ialbert, we'll chat about this today and get back to you on it. Thanks! |
@antgonza found a few possible approaches to scraping google groups: http://saturnboy.com/2010/03/scraping-google-groups/ |
I agree with @ebolyen that seeding Biostars with the QIIME forum google group may not be that effective. Based on @ialbert's description of how the import would work, I think we'll end up with a question per thread (which makes sense), but the first reply to the thread starter will become the one (and only) answer, and all following messages will be comments on the "answer". Often times, this "answer" will be a developer asking for more details, etc., so it isn't really an answer. I don't think we'll ever see a question with more than one "answer"; it will always be the first reply to the thread starter. Also, the answer will be polluted with a huge number of comments because these threads often contain a large number of messages. I'm worried that if we populate Biostars this way, we won't be setting a good precedent for how QIIME users should use Biostars to get help. Since we're fundamentally changing how we provide support, is it feasible to keep the current QIIME forum around in a read-only format and treat it as an archive? Some selective seeding of Biostars would help too. |
Second that, Jai. As a user, I don't want to see that old stuff. Plus so Andrew Krohn On Fri, Mar 27, 2015 at 8:24 AM, Jai Ram Rideout notifications@github.com
|
Perhaps this is too radical a suggestion but what if we limp along with the current forum until qiime 2.0 and make a clean break then? On Mar 27, 2015, at 8:26 AM, alk224 <notifications@github.commailto:notifications@github.com> wrote: Second that, Jai. As a user, I don't want to see that old stuff. Plus so Andrew Krohn On Fri, Mar 27, 2015 at 8:24 AM, Jai Ram Rideout <notifications@github.commailto:notifications@github.com>
— |
I favor Rob's idea. Seeding the old posts into Biostars does not seem trivial, so we could keep things under google forums for now when previous posts can still be useful for reference. Once we transition to Q2 I think those older posts will probably be less relevant |
I support @rob-knight's idea as well. A clean start matches the QIIME 2.0.0 design philosophy. Edit: The more I think about this, the more I like it. |
👍 |
Ok, I agree that this probably is the way to go (i.e., wait until QIIME 2). |
And in the meantime, we can start using the new BioStars for other projects, such as scikit-bio, biom, Qiita(?), etc? |
Sounds good… On Mar 27, 2015, at 11:46 AM, Greg Caporaso <notifications@github.commailto:notifications@github.com> wrote: And in the meantime, we can start using the new BioStars for other projects, such as scikit-bio, biom, Qiita(?), etc? — |
Thinking more about this... Since we get support questions for many of our different tools on the QIIME Forum, maybe we should create a general biocore forum on BioStars, and start directing the non-QIIME questions there. We can tag questions based on software package, and if any QIIME stuff does come up there we can tag as QIIME1 (ultimately vs QIIME 2). Google Groups is just so bad. Since it's so hard to find answers, we are just answering the same questions over and over. And we do need forums for skbio (SO probably isn't the most appropriate for this as we've learned by investigating it as an option for the QIIME Forum), Qiita, BIOM, etc. So in this case we wouldn't actively transition the QIIME Forum now, but start directing new non-QIIME questions to BioStars. We'd then colocate all of the biocore forums in one place using tags to differentiate, which ultimately would reduce effort because it's less to monitor, easier to link questions or tag as related to multiple projects, and it's not just dense threads of email discussions where it's impossible to tell correct answers from garbage (i.e., Google Groups). |
👍
|
Quick follow up on this. I got an updated ETA from @ialbert on BioStars 3, and it should be available later in the summer. He said that it's ready for release, but he's waiting until he gets back from vacation in August to release. I think that timing is fine for us. |
Starting at the Biocore 14.10 summit, we've been discussing the future of the QIIME Forum, and whether we should migrate from Google Groups. There are a few core motivations for this:
We discussed three options for doing this. The first was just pointing users to StackOverflow to ask their questions there. This isn't really the intended use of StackOverflow, and the maintainers frown upon that. The next was to use Biology StackExchange, but that is more for general biology questions, not bioinformatics technical support. Finally, we discussed using BioStars (web, paper), which is more directly intended for this purpose.
I've been discussing using BioStars with the site admin and senior author on the paper, @ialbert. He prefers that we don't just direct our users there as that could overwhelm BioStars since we have such a large community. He suggested two options by email - I'm paraphrasing here:
I think both of these sound like good possibilities, and it sounds like we may even be able to export previous discussions from the Google Group to initially populate the forum (is that accurate @ialbert?). With respect to option 2, I would think that instead of using
qiime.biostars.org
, we would start by usingqiime1.biostars.org
, which would give us a natural way to separate out support for QIIME 2, which will be very different. @ialbert said that this would work fine.Opening this up for developer and community feedback, there are a few specific questions that would be good to get input on.
Thanks for the input!
The text was updated successfully, but these errors were encountered: