Feedback?

[sckott]

rather than filling your inbox with another email, pinging you fine folks here:

@ethanwhite @cboettig @dfalster @dlebauer @ibartomeus @wcornwell

I'd love your feedback on this thing

What it is and why

Trait data seems to be one of the most locked down bits of the data ecosystem. Datasets are out there but often in supplemental materials, etc. that are hard to discover.

This service aims to be primarily an REST API that can be searched for datasets based on

• taxonomy
• location
• trait type
• general search across all fields

I got a Amazon research grant for $500 credits for https://github.com/sckott/gbids but it's not seeing much use as I thought it would get. So i'm trying to pivot to this idea, which i've been thinking about for some time.

There's description of the API at https://github.com/sckott/traitdb#api

• Dataset metadata is gathered, and is at https://traits.party/datasets/
• Individual dataset routes https://traits.party/datasets/z85a07642-9f49-408c-a16f-f71135d9450f give metadata for the single dataset
• /:datasetid/fields https://traits.party/datasets/z85a07642-9f49-408c-a16f-f71135d9450f/fields/ gives the fields in the dataset
• /fetch https://traits.party/datasets/z85a07642-9f49-408c-a16f-f71135d9450f/fetch retrieves the entire dataset from S3
• the /search route (not running yet) searches across all datasets
• the /:datasetid/search route (not running yet) searches in a given dataset

Both /search routes are based on Elaticsearch, and current plan has the entire datasets loaded into ES. I don't think this will scale for sure, so evaluating other options while still allowing searching datasets. The stuff returned from this route are the actual records (aka rows) of data from the datasets, so one can get just the records they want based on some search.

There's only 6 datasets in there now, as I evaluate what works and what doesn't, and get feedback for you :)

I haven't yet cleaned up/standardized these but will be useful once done:

• taxonomy
• geolocation (lat/long, or similar)
• place names
• other things?

Then we could allow search specifically on those elements instead of just a full text search across all.

There is no website for this yet, but could make one on top of the API, for users that prefer a GUI.

What do you think?

• Do you think people will use this?
• Any issues you see with licensing/etc? All Dryad datasets are CC0, and I've been using those so far. AFAIK supp. ESA jourjals datasets are CC licensed, so I think those are all fair game as well
• Would it be better to not actually serve datasets, but only be a discovery service?
• If this was to narrow focus, what would be of the most benefit to the most people?
• other things?

[wcornwell]

My two cents: I think we really need a discovery service. We need lots of things to be sure, but maybe discovery would be a good place to start. As the repositories multiply--Dryad, Figshare, Zenodo, GigaDB, USDA, ESA, Australia has it's own one now-- (see NPG's list). There is basically no way to keep on top of what's coming out. It would really useful for me if there was some kind of automated search for trait data. Is there some way to (smartly) automate dataset discovery?

[ibartomeus]

• Do you think people will use this?
  In my experience lots of ecologists will look for trait data in a web portal, few using R and almost none using other programmatic tools.
• Would it be better to not actually serve datasets, but only be a discovery service?
  I think a two step process 1) locate where the datasets are (that is search through dryad, etc...), 2) download them if you want, would be optimal.

https://traits.party gives a Forbiden message.

• Others: I am working on a curated open trait database (initially for bees). One big problem is the standardization of search terms, how do we call same traits, trait units, etc... Locating datasets with traits would be already a big improvement, but I imagine collating those traits in a single table would still need lots of tailored work depending on the sources. I'll keep you posted about my progress.

[sckott]

@ibartomeus

In my experience lots of ecologists will look for trait data in a web portal, few using R and almost none using other programmatic tools.

Sorry, should have included that it'd be straight-forward to make a website on top of the API for people that prefer that, and can easily use in R, or Python, etc. I'll edit my description above

[sckott]

@wcornwell Thanks for your thoughts.

Is there some way to (smartly) automate dataset discovery?

Perhaps. If it was a catch all dataset discovery that'd be a different thing, useful, but diff. from this (assuming this remains focused on traits). I think it could be done either using web services (e.g,. Dryad's OAI-PMH or Solr service) or just scraping if needed. In terms of user interface, could add RSS feeds, emails, other notifications about new items coming out.

[sckott]

@ibartomeus Sorry, the base URL is supposed to autoredirect to heartbeat, in most cases it does that. do no other routes work for you?

The bee dataset sounds great! Right, two problems: discovering datasets, and then merging, are both hard.

[ethanwhite]

1. I definitely agree that "trait data seems to be one of the most locked down bits of the data ecosystem" and that work any work to remedy this is useful.
2. For me personally access is more of an issue than discoverability, but I agree that they are both important. As such I'd prefer access to data as well.
3. How are you thinking of this as relating to TraitBank, which as I understand it is trying to solve the discoverability problem and also providing access to the raw data through APIs. Their website allows search on most of the things you listed and you've already handled access to the API in https://github.com/ropensci/traits. Is it that it's not ingesting enough of the available data? No longer being actively developed? Something it's missing?

[cboettig]

Re discovery, you might want to consider searching DataONE which already indexes metadata from m major ecological repositories

[cboettig]

Or maybe DataCite, though it might not expose enough metadata for very rich search, it would include the general purpose repos like figshare & zenodo. (Incidentally, that will be a weakness in general with repos like figshare where there is very little metadata to help discovery

[sckott]

@cboettig Right, for discoverability there is DataONE and DataCite - so we don't want to reinvent what they're doing. They aren't specific to traits though. But some subset will be trait datasets.

[sckott]

@ethanwhite Thanks for your comments.

access is more of an issue than discoverability

good to know.

How are you thinking of this as relating to TraitBank

The traits pkg could pull data from this API as well, so yeah, that's sorted. wrt Traitbank, AFAIK they don't really expose much of anything in their API for traitbank, that is I think you can only request for a given species ID. So you can't search Traitbank via the API, only via the web interface. I don't aim for this to be species centric as they are doing, where they're collating traits for individual species.

For this service, datasets will be the units of observation. Then we layer on top Elasticsearch to allow flexible query across datasets, so we can maintain the individual datasets as units.

Also, I think API first thinking will make this much more flexible and widely used in the end since it can be used in anything, whereas almost all Traitbank web portal functionality is not in their API (so not helping with reproducibility)

I have a feeling Traitbank is not running on a full tank of gas, but that's just my impression from activity on it and how long it takes to fix things.

I don't have a sense for how much data they have.

[sckott]

also love to hear from:

@dmcglinn @dlebauer @zachary-foster @davharris and anyone else really

[dlebauer]

Do you think people will use this?

Yes!

Any issues you see with licensing/etc? All Dryad datasets are CC0, and I've been using those so far. AFAIK supp. ESA jourjals datasets are CC licensed, so I think those are all fair game as well

Yes! with both licensing and attribution. It is an unsolved problem worthy of another thread and many conferences / working groups. CC0 makes it easiest to share data but makes it hard to give credit and track provenance. I'll punt on this after I apologize to anyone for the fact that it is hard to aggregate data and also cite every source (though we try).

Would it be better to not actually serve datasets, but only be a discovery service?

No!

There are many discovery services. The need for an interface that provides a consistent interface to many datasets is much more valuable than a discovery service. And there is a need for guidance to researchers on how they can prepare data to make it most easily reuseable. Consistent metadata only gets so far - consistent formats will do much more to facilitate synthetic research.

First, users of trait data would benefit from knowing how they can collect and arrange their data so that it can be combined with other data.

Second, the hardest part of using data from multiple sources is converting two formats into one. It doesn't need to be done by every person who wants to combine the same two datasets. But if it is done twice, we can use that as an opportunity for qc.

While any format must balance generality with utility at some point there may be a need for a different approach, e.g. to capture additional information. But we can move toward doing this more based on need than by lack of guidance. If we (community that creates and uses trait data) can converge on a countable number of clearly defined formats, it will be easier to develop translators and interfaces. This will make it easier to leverage each other's data in our own pipelines.

If this was to narrow focus, what would be of the most benefit to the most people?

Hard to tell because the API returns 'forbidden' (#2) but

I think the value will be in providing access to many datasets in a common 'homogenized' format. AND providing scientists with a template for their own data. Once the data formats (aka 'data models' or 'column names') are defined the value is in creating importer / exporter functions that translate among diverse data models (e.g. so that traitdb could easily import dumps from betydb and vice versa).

As an example I will talk about the database I've developed called BETYdb (on github and accessible via the ropensci traits package). BETYdb aims to solve some of the issues related to data homogenization and accessibility. BETYdb was designed with a web application that facilitates the collection of data extracted from figures and tables in journal articles, found in supplementary datasets, collected in the lab or field, or streaming from sensors. The primary portal betydb.org provides open access to the data it contains. But the web application can be used independently, and there is currently a network of ~10 independent instances of the database that sync their public data.

Back to the point about having a countable number of defined interfaces; if you can write a single interface between the betydb and traitdb APIs, traitdb could host any data shared by users of BETYdb. Furthermore, the data available in traitdb could be synced to all BETYdb servers, to be used in teaching and research.

This example extends to any other trait database - each one provides its own functionality and has its own user community. Effort should be made not just to have a single database but to allow all trait databases to share each other's data. In fact, genomic databases also contain trait data (e.g. to support analysis of gene-phenotype relationships using QTL and GWAS analysis). We are currently working on interfaces between BETYdb and genomic databases. These databases contain a lot of domain knowledge and functionality required for specific use cases. Coupling databases is the most efficient way to make use of and keep up to date with current functionality and user needs.

[sckott]

thanks for your feedback @dlebauer !

I get your point about talking to other databases - though first thing to focus on is what is this thing - happy to explore talking to other databases later. So yeah, what should this thing focus on? rolling over to next comment:

putting data into homogenized formats would be nice. I'd like to make certain fields across datasets homogenized to make searching across them easier (e.g.. ,taxonomy, geospatial, place names, etc.) and that should help some.

[sckott]

It seems somewhat narrowly focused databases are the most successful, as perhaps its easy to pitch to the target audience, and the interfaces/etc. can be made specific to the field, and curation/homogenization is tractable given the narrower scope.

• One idea is to focus entirely on supplementary datasets that are not in dryad/other repos - so rescuing datasets that would otherwise be very hard to find. Then we're not duplicating what's already searchable, etc. in dryad and friends - but does that focus not really serve a useful goal?