handling access authentication for datasets

[luisoala]

situation: many datasets are openly available but require some type of authentication (create account, etc..) before download is possible.

question: how can we let users of datasets know?

possible solution: allow creators of croissant datasets to add some "authentication_information" field where they write down in natural language how the authentication is done.

[marcenacp]

hey @luisoala, we support basic authentication (see: Loading a distribution via HTTP with Basic Auth).

on top of that, we could allow manual downloads in mlcroissant. Meaning you could input a hashmap filename->filepath on disk where you manually downloaded the data. that would allow to handle more complex use cases where an authentication is needed.

WDYT?

[stefanches7]

It seems that something like the HTTP authentication for "people in the wild" could be far-fetched. See attached an example from https://www.scidb.cn/en/detail?dataSetId=067dfc6234b140ef9e3ba7b91c27684d

export (4).json

unfortunately, everything the Croissant file specifies is

"conditionsOfAccess": "RESTRICTED",

and if you go to the website (not machine-actionable already here :( ), the page says "send email"

I wonder if there are some mechanisms that need to be put in place for cases like this. In Healthcare research it is also quite common that one gets access to the data only after a lot of paperwork (see UK Biobank, NAKO in Germany, IMAGEN etc.)

Thinking to give a Croissant WG presentation on the topic at this point. Curious to hear your thoughts too @marcenacp