OAI-PMH compatibility broken / Records assigned to more than one sets are not mapped correctly

[tkgesis]

Usually, if a record belongs to two oai-pmh sets, both the ListIdentifiers and GetRecords response of that record contains all the oai setSpec that this record is associated with:

https://easy.dans.knaw.nl/oai/?verb=ListIdentifiers&from=2012-06-01&until=2017-02-07&set=driver&metadataPrefix=oai_dc

view-source:https://easy.dans.knaw.nl/oai/?verb=GetRecord&identifier=oai:easy.dans.knaw.nl:easy-dataset:31153&metadataPrefix=oai_dc

In the dataverses https://dataverse.harvard.edu and https://dataverse.unc.edu/ we see a different behavior.

The oai ListIdentifiers for UNC for odum_all shows always only one set
https://dataverse.unc.edu/oai?verb=ListIdentifiers&from=2012-06-01&until=2017-02-07&set=odum_all&metadataPrefix=oai_dc
https://dataverse.unc.edu/oai?verb=ListIdentifiers&from=2012-06-01&until=2017-02-07&set=wilis&metadataPrefix=oai_dc

If you resolve a record via GetRecord, the setSpec differs from the one indicated in the ListIdentifiers :
https://dataverse.unc.edu/oai?verb=GetRecord&identifier=hdl:1902.29/10438&metadataPrefix=oai_dc

The multi-valued setSpec relationship is apparently not mapped completely. It is always only one setSpec, and which setSpec is shown varies upon which oai-pmh verb you use.

[donsizemore]

Full disclosure: Odum aka dataverse.unc.edu hasn't yet upgraded to 4.6 due to some illness and a pending vacation. We hope to upgrade in late March to pick up the harvesting fixes included in 4.6.

[pdurbin]

@tkgesis thanks for the bug report. Are you saying that in Harvard's Dataverse and in UNC's Dataverse you are seeing the same behavior? As @donsizemore points out, they are running different versions, v. 4.6 build 62-c913662 and v. 4.5 build 41-dc58ae1 respectively (and we'll be updating Harvard to 4.6.1 soon).

[tkgesis]

@pdurbin you are welcome. We saw this behaviour a while ago on Harvards Dataverse, too. I don't have an example URL at hand.
The issue could root in the fact that in dataverse, oai sets are defined as queries, rather than sets.
Is the assignment of a record to a set (i.e. record X matches set queries a,b...) persisted or cached somewhere?

[landreev]

Yes, it's important to re-test this in 4.6. Numerous OAI server improvements were made between 4.5 and 4.6. The OAI support in 4.5 was still somewhat limited (it wouldn't pass the OCLC standard validation test; the 4.6 implementation does pass).

The assignment of record to a set is persisted (search queries define what goes into a set, true; but it's not being done in real time, every time a set is accessed - the results are saved when the set is created/updated).

That said, I don't specifically remember checking/testing this... And I'm pretty positive our own, Dataverse harvester does not rely on this information for any practical purposes. I.e., when we request ListIdentifiers for a specific set, we never check the set assignments of the individual listed entries. In other words, I would not be entirely shocked if what you reported turned out to be true, even in 4.6. Should be easy to fix though...

Out of curiosity, what is the practical purpose of having all the sets fully listed in individual records, in your case?

[tkgesis]

The assignment of record to a set is persisted (search queries define what goes into a set, true; but it's not being done in real time, every time a set is accessed - the results are saved when the set is created/updated).

The set - record relationship is 1:n. Are you saying, that whenever any of the defined sets is updated, the set assignments for each record are updated? Without any knowledge of the dataverse code, I'd guess that the set assignment need to be set at two times:

1. after create/update of a new record and
2. after create/update of a new set query, for all records in the latter case.

Out of curiosity, what is the practical purpose of having all the sets fully listed in individual records, in your case?

The purpose is

1. to enable more efficient harvesting, less network traffic and less need for storage and
2. keep http:\datasearch.gesis.org free from duplicates.

Let's assume a record is assigned to two sets. With an OAI-PMH compliant ListIdentifiers response, clients can determine, e.g. by the identifier, if this record is already present, and therefor skip the GetRecords request for that particular record.

OAI-PMH as a standard and a means to abstract domain specific representations into generic ones makes it possible for the harvesting side to re-model the (hierarchical) categories without any knowledge of the underlying system, which could be dspace, dataverse, da|ra or other.

See also

[landreev]

Regarding records-to-sets assignments and set updates:

Yes, it's a 1:n relationship. And yes, your description of how the updates are supposed to work appears to match how it's actually implemented in Dataverse.
Every time a set definition is explicitly updated, it triggers a full rebuild of the set. Extra records will/may be added to the set; records that should no longer be in the set will be marked as "deleted" from the set. This set update will NOT touch any entries belonging to any other sets.
In addition/in parallel to this, OAI records are updated whenever a specific dataset has been updated by its owner(s). This update WILL update all the, potentially multiple, instances of this record in all the different sets.

So, once again, if records belonging to multiple sets are not getting these multiple sets listings in ListIdentifiers/ListRecords, it should be a straightforward fix - since we definitely have the information saved in the database.

I'll test this in our (harvard/iqss) production today, and will confirm what the behavior is in the latest (4.6) OAI implementation.

[landreev]

With an OAI-PMH compliant ListIdentifiers response, clients can determine, e.g. by the identifier, if this record is already present, and therefor skip the GetRecords request for that particular record.

With the records you get from the Dataverse OAI-PMH implementation, you can check for duplicates based on the identifier alone - as we guarantee them to be unique across all the sets available. Whether you can/want to rely on this for other OAI archives is another question of course... But, honestly, so far I can't exactly see why not. I do understand that it is not necessarily the case/requirement, that records for the same object that appear in multiple sets must have the same exact identifier. (I've seen OAI archives that embed the set name into the identifier... thus making them NOT unique). HOWEVER, in your case, for you to be able to make the decision to skip the GetRecord call it MUST have the same identifier across all these multiple sets... otherwise having the multiple sets listed isn't going to help you - because, how would you know that it is the same thing exactly?

I may still be missing something, but it appears to me that for the purposes of knowing whether a record you are harvesting is a duplicate or not, it may be less important to know what sets it belongs to on the OAI server side; because the only thing that is important is whether you have already harvested the record or not... And for that you have to rely on the identifier... - ?

[landreev]

Another interesting issue this brings up, what do you do when you see a record advertised in ListIdentifiers?set=foo... as "deleted" ?
Because in the case of our implementation, this really means that the record was deleted from the set "foo", but not necessarily from any other sets. As a result of, for example, the change in the set definition.
Do you assume that this means that the object was deleted from the remote archive, "for reals"?

[landreev]

Finally - and sorry to go tl;dr on you - I have actually assumed that in real life nobody would be harvesting multiple sets from us. Specifically in the Odum case, they appear to have created an "everything they have" set - and that's presumably what you want? (do you harvest other subsets as a way to preserve some individual collection hierarchy... ?)
That assumption may have been entirely incorrect - so that's why I'm genuinely interested in your opinion, as an actual real life content harvester.

[tkgesis]

Thanks for the explanation in how you handle deletion.

We assume the 'deleted' flag as referring to one single record, identified by its unique identifier.

If the record carries the 'deleted' flag, we remove it from the http://datasearch.gesis.org index.

We do recognise changes of the set assignments of a record, given that the all sets appear in the ListIdentfiers response.

The odum case is a good example for selective harvesting.
Odum_all contains more than we need, and if we harvest the other sets, not everything we'd like to include is covered by those queries aka sets.
Apart from the mentioned fact that they contain only one setspec that does not correspond to the one we sent with the OAI-PMH request.

Datasearch.gesis.org facilitates search across datasets in the social sciences. Therefore we carefully select the institutions and sets for harvesting: the topic is social science and the record should denote a dataset rather than articles and the like.

Set level:
We review the records of each set and decide whether a set focuses on social sciences. If so, we harvest the entire set.

Record level:
The second distinction, whether a record refers to datasets or text, is done by filtering. We automatically skip records that have a certain value in the dc:type field.

It would be great if the integrated dataverse oai server could be compliant with respect to the set assignments.

[landreev]

OK, I just checked, and, unfortunately, the latest (4.6.1) behavior is still NOT what you want. For example:
https://dataverse.harvard.edu/oai?verb=ListIdentifiers&set=UVA_Authored_Datasets&metadataPrefix=oai_ddi

<ListIdentifiers>
   <header>
      <identifier>doi:10.7910/DVN/RTJ38N</identifier>
      <datestamp>2016-10-27T18:35:56Z</datestamp>
      <setSpec>UVA_Authored_Datasets</setSpec>
   </header>
...

this record however belongs to 2 sets: "UVA_Authored_Datasets" and "IQSS".

So, it's official: we only report the setSpec that matches the requested set.
As I said, the fix would be trivial - but it will have to be in the next release.

[4tikhonov]

@landreev, we're also experiencing some problems with identifiers in OAI-PMH at the moment.
https://dataverse.unc.edu/oai?verb=GetRecord&identifier=hdl:1902.29/10438&metadataPrefix=oai_dc

We have harvester for narcis.nl research portal and it's not resolving identifiers (to http) for records harvested from DataverseNL as http missing in the current OAI-PMH output.

We think that

<dc:identifier>hdl:1902.29/10438</dc:identifier>
should be changed to
<dc:identifier>http://hdl.handle.net/1902.29/10438</dc:identifier>

Is it possible to fix that?