Inconsistent data between the harvesting server and client

[JacekChudzik]

After harvesting datasets we are finding differences between client and server. This mainly consider dates.

When I check api by calling /api/datasets/export?exporter=dataverse_json&persistentId=doi%3A10.57903/UJ/RH0J5L

I can see proper data in fields like lastUpdateTime and releaseTime.

However, the Dataverse UI shows the dates related to the harvesting moment.

Those wrong dates I can find in db by SELECT * FROM dvobject WHERE dtype = 'Dataset' AND authority = '10.xxx' AND identifier = 'xxx'

What is more sometimes even the title is wrong but with the same pattern: dataverse_json show correct values, but wrong one are stored in db and displayed to user.

This is confusing and should not happen. Problem is not related to dataset versions.

• When does this issue occur?

This is a bug related to harvesting and was noticed in the context of dates displayed to Harvester users.

When trying to fix by deleting harvesting client/server and reharvesting data from Dataverse collection - all datasets received one date on harvester: current date (as mentioned above: misleading dates are in dvobject as lastUpdateTime and releaseTime - when I correct them manually in db, then they are correctly displayed for end user)

• Which page(s) does it occurs on?
  Harvester pages visible for users.

• What happens?

Harvested data are incorrect.

• To whom does it occur (all users, curators, superusers)?

all

• What did you expect to happen?

harvesting client and server should display same data for users.

Which version of Dataverse are you using?
6.7.1 at this moment.

Screenshots:

dataverse_json:

displayed on harvester page:

[pdurbin]

@JacekChudzik thanks for opening this issue and for the discussion on the mailing list!

I went to https://rodbuk.pl/dataverse/root/?q=RH0J5L thinking I'd see the wrong title (with "upon stress" in it, like in your screenshot) but the title looks right:

Did you fix it manually? 🤔 Or am I confused? 😅

[landreev]

Dataverse definitely uses some special logic to display the dates on search cards for harvested datasets (that is different from what it does for local, non-harvested datasets). It is possible that it ends up showing the date of harvesting and importing, instead of the real publication date imported from dataverse_json.

The original post on the list also mentions that

... What is interesting, in harvester facets serach under 'Depist date' datasets are correctly categorized

i.e., it appears that the harvesting Dataverse does have the correct date in the database; so there's a good chance that the fix will be very simple and limited to the display logic for the search card.

I will take a look.

[landreev]

P.S. The date shown in the search cards on the harvesting, client Dataverse is not even the harvesting date, but the OAI timestamp - i.e., the lastupdatetime from the oairecord database table on the server side. Which is the timestamp of the last update of the OAI record itself; so, rebuilding the OAI set from scratch will in fact make all the records have today's timestamp.

From what I can see, the fix will be slightly more complicated than simply the display logic. I.e. the bug appears to be in the dataset import, where it defaults to the OAI timestamp for the releasetime of the dataset version, even when a real publication time is available in the json metadata. But should still be fairly straightforward.

Why you are seeing different titles in the search cards (intermittently??) is still a mystery to me. (Is there a chance that a different metadata format - oai_dc or oai_ddi - was used in the harvesting runs that resulted in a different title showing?)

[JacekChudzik]

As for the title in the indicated dataset - I haven't changed anything here and I don't understand why it's displaying correctly now. I'll investigate. If I come across any other instances of title differences, I'll let you know.

Since this issue was created I have made updates in harvester db but only regarding the dates.

To have a temporary workaround I have prepared a script that downloads the latest update and release date (via api) and then updates the harvester postgres by performing UPDATE datasetversion SET lastupdatetime = 'yyy-mm-dd 00:00:00' WHERE dataset_id = xyz AND versionstate = 'RELEASED'; - and it works quite nice, dates on server and clinet seems to match (ofc after reindexing).

[landreev]

Hmm, it really looks like this is the fix: 69b035c - i.e., it was due to a simple typo in the parser.
Dataset original:

On the harvesting dataverse:

Let me test some more, and also review the logic for the dates when the 2 formats other than dataverse_json - oai_ddi and/or oai_dc - are used for harvesting. Ideally, we should only use that OAI timestamp as the last resort, when nothing better is available.

It is objectively too late to be able get a fix for this into the 6.11 release. But I would be happy to build a patched war file for you, if you want.

I am a bit surprised that we haven't had more complaints about this.

[pdurbin]

@landreev thanks for looking into this. There's another issue about harvesting dates here:

• Usage of datestamp from Record during harvest using oai_dc format #8693

I'm not sure if your commit above relates to it or not.

[landreev]

Oy, 2022. It is a related issue. My commit above is not going to address it, since it fixes the bug specific to dataverse_json. But as I said earlier, I was going to review what we do with dates when harvesting via oai_dc and oai_ddi as well.
(it is objectively more difficult with oai_dc, since there is not much metadata to work with there; but it should still be possible to do better than what we are doing now, at least in some cases).

[JacekChudzik]

@landreev I'm glad you found the source of the problem and fixed it. Thank you for the offer but we will slowly crawl up o patched version. Our policy is to be 2-3 version behind the latest one so until we get there I'll use my python script to do the synchronization.