Prepare for Dataverse v4.0 compatibility

[leeper]

Compatibility guide (archived)

Harvard Demo server running v4.0

API Guide

[pdurbin]

Thanks for creating this ticket @leeper ! A few things...

Per http://guides.dataverse.org/en/latest/api there is an additional server I've set up specifically for API testing at https://apitest.dataverse.org where I deploy the latest code (more often than on the "demo" server), run some post-deployment tests, and share the results. It might be a better place for testing.

I also wanted to point out that a new Dataverse API Community Group is forming at http://community.dataverse.org/community-groups/api.html and everyone is welcome to join!

For my own reference, I'll include a link to an internal ticket where we've had a bit of discussion about backwards compatibility of APIs.

Finally, I sort of consider "DVN" to be a deprecated term. It stood for "Dataverse Network" and referred to software such as DVN 3.6. We're now dropping "Network" and calling the software "Dataverse 4.0". I'd be curious to hear your thoughts on the possibility of changing the name of the R package. I'm not trying to complicate things for you and of course it's fine to continue calling it "dvn", but I just wanted to put a bug in your ear about this! :)

Thanks so much for all of your hard work creating and supporting this package! We really appreciate it!

[leeper]

@pdurbin Thanks for the updated info. This has been on my backburner for a while, so it's helpful to get a status update. My current thinking is to launch a new package, so "dvn" will remain the package for using Dataverse < 4.0 (for backwards compatibility) and a new package with a new name (maybe just "dataverse"?) will be tailored around Dataverse >= 4.0.

I'm also thinking about writing a separate SWORD package (as we talked about a long time ago), so the new "dataverse" package will simply import that generic functionality.

Let me know how that all sounds to you.

[pdurbin]

@leeper if you start a new R package called "dataverse" please let me know! @cchoirat has been asking me about an R package that's focused on Dataverse 4.0 and higher. (Separately, I've been chatting with @rliebz and @raprasad about the possibility of a Python version.) If it's not too much work for you to maintain both packages, I think it's a great idea. Also, if you think it might make more sense to put the new "dataverse" package under the @IQSS GitHub organization, please just say the word.

SWORD spec lead @richard-jones seemed very enthusiastic about a separate SWORD package for R and back in late 2013 he and I were hoping that someone could build off your work. If you're now saying that you might be able to do this work yourself, so much the better!

[leeper]

@pdurbin Okay, let me put some thought on this and I'll get back to you.

[pdurbin]

Heads up about IQSS/dataverse@3b57027 and http://datascience.iq.harvard.edu/blog/dataverse-40-next-week ! :)

Sounds like you're making good progress! If there's anything you need, just let us know!

I recently started putting API-related integration tests right in our main repo: https://github.com/IQSS/dataverse/blob/8496b4235bedd83aa1014bc9731b324f71478072/src/test/java/edu/harvard/iq/dataverse/api/SearchIT.java

Finally, @rliebz has been working away in this branch in Python if it's of interest: https://github.com/rliebz/dataverse-client-python/tree/4.0

[pdurbin]

The news on this end:

• new page on client libraries (including this one!): http://guides.dataverse.org/en/latest/api/client-libraries.html
• https://github.com/IQSS/dataverse-client-python how has Travis CI tests: https://travis-ci.org/IQSS/dataverse-client-python (passing even) :)
• we tagged Dataverse 4.0: https://github.com/IQSS/dataverse/releases/tag/4.0
• I recently posted this: [sword-app-tech] Should we extend the Repository API beyond SWORD?
• I mentioned this very issue Prepare for Dataverse v4.0 compatibility #23 today: https://groups.google.com/forum/#!topic/dataverse-community/fFrJi7NnBus :)

If there's anything you need from me, please let me know! As always, thanks for all you do for Dataverse! :)

[leeper]

Thanks for the updates! This is coming along, but I think what my current release plans are as follows:

1. Release new R client here: https://github.com/leeper/dataverse with minimum support for the search API, the data access API, and (hopefully) the native API. This will import the UNF package so that users can verify datasets locally against the metadata provided by the Dataverse.
2. Release a generic SWORD client here: https://github.com/leeper/SWORD. This will aim to work for Dataverse, Archivematica, and arXiv.
3. Import the SWORD client into the dataverse package and write some custom wrappers (essentially rebuilding the entire deposit workflow from dvn). (Programming style note: I think it will use reference class semantics from R6 rather than the function approach I used in dvn.)
4. Deprecate dvn and point new users to dataverse.

[pdurbin]

Release new R client here: https://github.com/leeper/dataverse

@leeper how do you feel about https://github.com/IQSS/dataverse-client-r as a URL? This would mirror https://github.com/IQSS/dataverse-client-python and I would be happy to create that repo and give you push access like I did for @rliebz for the Python equivalent! I can also enable Travis for you. Whatever you want. :) There are developers at @IQSS such as @cchoirat who have been hoping to see an official R package for Dataverse under the @IQSS GitHub organization. And maybe she'll be able to help out. ;)

No matter what the final URL is, I'm quite happy to update http://guides.dataverse.org/en/latest/api/client-libraries.html when the Dataverse 4.0-compatible version is available! As you mentioned, a pointing users from the old "dvn" package to the new "dataverse" package will be quite helpful!

[leeper]

This sounds like a plan. It's on my agenda for August (I'm moving on the 12th and will try to put it all together the week after that).

[pdurbin]

@leeper great! Sorry, when you mentioned IQSS/dataverse#2407 (comment) did you mean that you'd like me to create a repo under @IQSS for you? I'm happy to do so! Just say the word! To match https://github.com/IQSS/dataverse-client-python I might call it dataverse-client-r unless you have a name you prefer.

[leeper]

@pdurbin Yup, let's do that! dataverse-client-r sounds great.

[pdurbin]

Ok! I created https://github.com/IQSS/dataverse-client-r and gave you push access.

git remote add origin git@github.com:IQSS/dataverse-client-r.git
git push -u origin master

[leeper]

@pdurbin Thanks!

[leeper]

@pdurbin I don't think I have write access to https://github.com/IQSS/dataverse-client-r. I tried to push but got a failure message. Can you take a look and make sure permissions are configured for me?

[pdurbin]

@leeper sorry! I had created https://github.com/orgs/IQSS/teams/dataverse-client-r but apparently not added that team to the repo. Can you please try again?

[pdurbin]

@leeper have you had a chance to re-try pushing to https://github.com/IQSS/dataverse-client-r ? I think the permissions are set up correctly now.

I'm asking because someone just asked on the Dataverse mailing list was just trying to use the old "dvn" package with a Dataverse 4 installation (which won't work, of course): https://groups.google.com/forum/#!topic/dataverse-community/0RmQ7ovfhDs

Thanks again for all the work you've done for Dataverse!

[leeper]

@pdurbin Yes! Just pushed an initial draft successfully. There's no documentation yet and it's untested, but it theoretically works. Will make an effort to document and add tests as soon as I can.

[leeper]

Closing this, as development for Dataverse >= v4.0 is now here: https://github.com/IQSS/dataverse-client-r