statementDate not updated on republish

[tiredpixel]

When statements are migrated, for example in order to add LEIs (openownership/register-sources-oc#8) or to fix identifiers (#189), it is necessary to republish them. Doing so results in a new statement with a different statementID, the old statement referenced within the new statement's replacesStatements, and metadata.replaced set for the old statement. This much seems fine. However, statementDate is not updated, and the old statement's date is maintained despite the changes. Is this correct behaviour?

BODS 0.2 and BODS 0.3 say:

Statement date
The date on which this statement was made.

But this does not help much. When is the statement 'made'? Originally in the third-party datasource (e.g. PSC), despite being supplemented by other data (e.g. OpenCorporates), meaning existing behaviour is correct? Or when that statement was generated, meaning existing behaviour is incorrect and statementDate should be updated to today?

[tiredpixel]

If it transpires that statementDate should not be updated when republishing statements (i.e. changed statements), then I recommend that Register implement its own internal extension to BODS to track this publishing time, similar to existing metadata.replaced, which is necessary to performantly track replaced statements and accurately paginate. Trying to develop or debug a chain of statements where everything looks almost identical and no dates have changed despite some things having just been republished is very time-consuming.

[tiredpixel]

I've only just noticed publicationDetails.publicationDate, which does appear to be updated. So perhaps the current behaviour is correct—i.e. statementDate should not be updated, but publicationDetails.publicationDate should be?

References #230 , during which this was discovered (and also containing the explanation for why it was not discovered sooner).

[kd-ods]

Given that the source of statements is given as the register to which the info was disclosed, I think you're right that "statementDate should not be updated, but publicationDetails.publicationDate should be".

[tiredpixel]

Thanks for the clarification, @kd-ods .

Could I ask whether BODS 0.4 plans to allow publicationDetails.publicationDate to be a datetime, not just a date? I believe both BODS 0.2 and 0.3 say it is a date.

If there are no plans, I propose that metadata.publishedAt or similar is added to Register to track this more granularly, to make data exploration and debugging much easier with regards to Elasticsearch. It would also lend itself naturally to a choice of Kibana timestamp column, so documents could finally be ordered in the interface. This would only affect Register's implementation, though, and wouldn't require any changes to BODS (similar to existing metadata.replaced to make replacesStatements performant).

[kd-ods]

Could I ask whether BODS 0.4 plans to allow publicationDetails.publicationDate to be a datetime, not just a date? I believe both BODS 0.2 and 0.3 say it is a date

Yes, it will be a datetime. (It's been worked on as part of openownership/data-standard#162.)

[tiredpixel]

That's great, @kd-ods . Okay, then I think we can close this ticket without action at present. Thanks for your help.