Set up website + document updates automatically

[rhine3]

As discussed in #1 - it would be great to create a filterable website (and possibly an automatically updated README?) for the bioacoustics software list. @NickleDave suggested potentially basing this off of the system that @vsoch used to create the Research Software Encyclopedia.

[NickleDave]

Any chance I could join in if/when you and @vsoch meet about this, @rhine3?

I would like to set up something similar for https://github.com/vocalpy/pollyglot and it might help to hear you both talk it out.
Not sure if I can do more than get in the way but also happy to pitch in on dev stuff if I can be of service

[vsoch]

That would be ok with me! I pinged @rhine3 via email end of June but no response yet, so likely busy. If you want @NickleDave we can also just meet to spec out what you have in mind first - whatever the order turns out to be one can definitely help Toni fluency the other!

[NickleDave]

👍 thank you @vsoch -- I think @rhine3 might have been hanging out with some birdies this last week and will probably be back on-line presently.

My project is definitely low on the to-do list, buried underneath a ton of neglected maintenance work on other projects.
Also to be honest I am hoping to learn from how this goes 😈

[NickleDave]

tagging in @louisfh so @vsoch can tag us all later

[vsoch]

Issue opened! rseng/rse#59 It's on my TODO list so hopefully will start this weekend or an evening next week. Thanks y'all for meeting it was fun, and keep me in the loop!

[vsoch]

hey folks! I started the export UI this weekend! Based on our discussion I'm assuming the main design should have a searchable table - but do y'all have other preferences? Point me to sites you like the design of? I'll try to make it customizable (via the jekyll template) but ideally I'll start with a nice layout / theme that most people like.

[vsoch]

okay looking good so far! I have added an export functionality (no PR yet, still tweaking) and started a template with a nice home page for a description / logo, and very basic software page with a table (akin to the rseng software but with annotation fields removed).

I've also done search...

and a very basic json API to get metadata for all the software in the database:

I think the average site is going to maybe want news (e.g., for posting news or blog articles) and then an about page - and a quick question for y'all - do you want each collection entry to have its own page, and if yes, what would you like it to look like? It could literally be another table with more metadata (or something else)... I wanted to ask first before doing anything.

Also I found this kind of cool editor template that we could use in some cool way with the local cache - my dummy example was to install things into a custom made Dockerfile, but you could imagine just navigating around the site and making a custom list of software to export? Do something else with? Let me know your thoughts!

Still lots to do, but I promised I'd keep you updated! Will post again when I have more (and of course let me know your thoughts to the above!)

[vsoch]

Update - vision for future! When we have more organized tags, let's add a visual like this -> rseng/rse#60

[vsoch]

ooh this looks pretty cool! This is unstructured GitHub tags (will come with default)

[NickleDave]

ooh pretty

[louisfh]

Cool visualization for tags - great work so far, thank you Vanessa! I think a simple clean table looks good. I like the visual style of Elicit (which is also a cool tool worth checking out).

[vsoch]

@louisfh do you know what table framework they use under the hood? I can't see an actual table without an account I don't think. Basically if you view the source and look at the javascript and javascript libraries - does a table framework jump out?

[vsoch]

Some more tiny progress! I made a dummy spreadsheet tonight and have the basic import done:

The reason there are only 8 is because I'm requiring fields for a Url, Title, and Description, and the majority in the sheet don't have a formal title. I might just allow that? Haven't decided yet. I do think having some minimal information for a person to parse is pretty important (e.g., a title and description with a URL seems like a good set!)

I just need to finalize the import structure for a "custom" parser entry (meaning it's not from GitHub or GitLab) and then I should be able to add this feature, and continue on to the next! At the end I'll be able to make a custom database from your spreadsheet, and then have it update a site nightly along with re-import (which I'll write workflows for).

[vsoch]

Okay import from Google sheet is done:

• PR: https://github.com/rseng/rse/pull/63/files
• docs: https://rseng.github.io/rse/getting-started/commands/#import

[vsoch]

Alrighty! So I think I've added the major features we discussed? https://twitter.com/vsoch/status/1548443989539377153 I will still look into the extra for run citation parsing, but for the rest (addition of a schema / update of the analyzer) I think I'll probably need y'alls help since I know nothing about this space! For easy access here is the issue we were following: rseng/rse#59. And here is a complete thread with updates: https://twitter.com/vsoch/status/1548443989539377153. I wound up making a demo sheet that others could copy as an example to import from csv.

And nooo rush!! I wanted to get this ready for you asap, and I'm here when you need me for the rest :) @NickleDave there should be enough docs in the example site if you wanted to give it a go for your own project, and of course give feedback or ping me for any wonky JS/css/html help. I will update again if I find out anything interesting for parsing Google Scholar / references (never tried it before!)

[NickleDave]

@NickleDave there should be enough docs in the example site if you wanted to give it a go for your own project, and of course give feedback or ping me for any wonky JS/css/html help.

Excellent thank you @vsoch! -- will do

[NickleDave]

Hi all,

I have a branch in my fork where I was trying to automate generating the database + serving a site:
https://github.com/NickleDave/bioacoustics-software/tree/add-rse

@vsoch I was able to get rse import to work (very nice! I didn't know you could do make export links with Google sheets like that--much easier than dealing with OAuth). But when I run rse start I don't see any listings.

Also re: serving the site:
it looks like the rseng repo has two branches, 'master' with a Python package "backend", and the gh-pages branch that's the actual site (https://github.com/rseng/rseng/tree/gh-pages).
How do they talk to each other? I assume magic happens in GitHub Actions somewhere?
Is there documentation somewhere on how to serve a site from the output of rse?

Thanks!

[vsoch]

@NickleDave you don't need rse start, that's for the flask server. You just need to generate the database, and then copy this repository to grab and update from it. https://github.com/rseng/web

[vsoch]

Or just copy this entirely :) https://github.com/rseng/rse-jekyll-web/

That repository README has full instructions.

[NickleDave]

Or just copy this entirely :) https://github.com/rseng/rse-jekyll-web/

🙌

Thank you!

Is there a link to that in the rse docs somewhere? (Do you want a PR?)

[vsoch]

Yeah that would be great! So the specific import/export are documented on the commands page https://rseng.github.io
/rse/getting-started/commands/index.html#export and I think perhaps we should provide a link to the repository I shared above here: https://rseng.github.io/rse/tutorials/index.html. Thoughts?

And yes, would love a PR!

[NickleDave]

👍

So the specific import/export are documented on the commands page https://rseng.github.io/
/rse/getting-started/commands/index.html#export

I did find the command although finding "commands" itself was not obvious to me.

I mainly meant adding a link to the repository you shared.

I think perhaps we should provide a link to the repository I shared above here: https://rseng.github.io/rse/tutorials/index.html.
And yes, would love a PR!

👍
I'll open an issue on that repo to not add noise here

[NickleDave]

Reporting back here that I was able to generate a static site by just doing rse export.
Not quite clear to me if this is the "right" way to do things, e.g. if we want to automate updates of this site.
Discussion here: rseng/rse#71

I also tried generating a full database for the site using a copy of the entire spreadsheet but hit a bit of a bug there.
Discussion here: rseng/rse#73

[vsoch]

Tweaks to add links to tutorial, and bug fix rseng/rse#75

[NickleDave]

Hi all, just wanted to update you where I'm at with this:
https://github.com/NickleDave/bioacoustics-software/commits/add-rse

Thanks to a ton of help from @vsoch I am at least able to build a site:
https://nickledave.github.io/bioacoustics-software/software/

Tada, 🎉

This is the copy of @rhine3's sheet that I am using to be able to do this
https://docs.google.com/spreadsheets/d/1Ba1MY4o5Sm1f08IekJcbxAtSjkDN71Z1RZ42kzrofJ0/edit#gid=0
basically following the README here:
https://github.com/rseng/rse-jekyll-web

The main issue right now is that a bunch of the links in the 'url' column are "malformed" according to the rse parser. @vsoch made some fixes so we can at least import without completely failing. I'm not quite sure how to fix those urls at the moment.

@vsoch you have done so much work already that I am afraid to suggest this, but I wonder if it would make sense to add an importer/scraper that can take a .csv directly?

This way the data could be munged + cleaned, e.g. by a script, before handing it off to rse.
@rhine3 and I had thought we might need to do this anyway.

I have a vague idea how this might be possible in rse, will open an issue there to suggest

[vsoch]

yep happy to do that.

[rhine3]

Fantastic work on this @NickleDave!! Thanks so much to both you and @vsoch for making this project happen. Sounds like the finish line is in sight 👀

[vsoch]

All set! rseng/rse#81

[vsoch]

Included in release https://pypi.org/project/rse/0.0.46/.

[NickleDave]

Hi @rhine3 I went ahead and made a PR for this.

@vsoch very patiently worked through a couple minor hitches with me but we have got it going now.

If you were to merge now it would add the site, but generated from a copy of the spreadsheet I made, and still with the original RSEncyclopedia template (:thinking: forgot about that part, we probably need to address that).

I think we could just get it to generate directly from your spreadsheet if you were open to doing the following:

• remove the top row that is not a header ... I just clicked on the link again and it seems like you might have done this already?
• change a couple of urls that have GitHub in them but are not pointing to project/repos. E.g., change "https://rflachlan.github.io/Luscinia/" -> "https://github.com/rflachlan/Luscinia", so the parser doesn't get confused and so the scraper can get metadata off that repo directly

If you're willing to make those changes, then we can change the url in the GitHub actions file to point directly at your spreadsheet and not have to worry about cleaning up the csv manually. After the fixes @vsoch made we were able to parse all 70 entries.

Let me know how to proceed. Happy to close the PR and make a new one later if we need to backtrack a bit.

[NickleDave]

and still with the original RSEncyclopedia template (thinking forgot about that part, we probably need to address that).

Actually I'm lying. We just need to change the logo and text in the template as the very clear instructions that I did not read tell us to.

https://nickledave.github.io/bioacoustics-software/

Welcome! This is your custom interface to display your research software database. You can see the database table here. Customize the logo to the left in the "_config.yaml" and this text in index.html. You can include a link like this. This description spot we recommend adding some detail about your database, and important links below.

[rhine3]

David, I made those updates to the spreadsheet and created a website-dev branch, so would accept a PR into that branch from your addRSE branch whenever you're ready! Thank you so much!

From there I'll work on the following:

• Change URL in GitHub actions file to point directly at spreadsheet
• Change logo and text in template, including adding information about a Google Form to submit info/corrections

[NickleDave]

Yay 🎉

would accept a PR into that branch from your addRSE branch whenever you're ready!

Done!
I changed the "base branch' of the existing PR to your website-dev branch. (I think I did that right, haven't done this before)

[rhine3]

Hi @NickleDave and @vsoch -

I was able to make those two updates listed above. There are definitely a few things left to update, but I'm trying to give this repository a DOI ASAP as I am planning to reference it in a paper!

So here's my remaining to-do list for accountability:

• Remove "News" section
• Add a "most active" star or replace the News section with a "Where to start" section and reference this on the "Software" page
• Add a more detailed "About" page
• Update README
• Fix up columns/auto-generated tags
• Create DOI and add to README
• Update Favicon
• Make sure tags page auto-updates