Remove preview file values

[benjamingeer]

This finishes the work needed for issue #1068. Since PR #1011, preview images in the triplestore and in Sipi are obsolete. This PR removes them and changes knora-base accordingly.

• knora-base.ttl:
  • Remove everything about preview images.
  • Use owl:cardinality 1 on knora-base:hasFileValue and its subproperties, as per Adapt image handling in API v1 to use IIIF for previews #1068 (comment).
• Change API v1 and its Sipi scripts (convert_from_file.lua and convert_from_binaries.lua) so that they no longer store preview images.
• upgrade/README.md: Add a simple convention so Knora release notes can provide scripts and instructions for updating existing data when Knora is upgraded.
  • upgrade/1230-delete-previews: Add scripts for updating data for this PR.
  • upgrade/1211-datetime: Add a script for updating data for PR Change all xsd:dateTimeStamp to xsd:dateTime in the triplestore #1211.
• Remove preview images in test data (using a transformation added to the TransformData utility).
• Update docs and release notes.

Resolves #1068.

[benjamingeer]

@tobiasschweizer Could you please try updating the BEOL data using the scripts in upgrade/1230-delete-previews, and let me know if you find any problems?

[tobiasschweizer]

@benjamingeer Ok, I will try it. I think I will be able to do that tomorrow.

[tobiasschweizer]

I think the Python load test script would also have to be adapted. It is very similar to the import process I am using.

[benjamingeer]

I think the Python load test script would also have to be adapted.

Which script is that?

[tobiasschweizer]

Which script is that?

It is the load test script written in Python. @subotic knows where it is.

[tobiasschweizer]

I assume you are getting rid of the non gui case (shared storage of Knora and Sipi) so the import process for StillImageRepresentation has to be adapted.

[benjamingeer]

It is the load test script written in Python.

I didn't know that such a script existed.

I assume you are getting rid of the non gui case (shared storage of Knora and Sipi)

I haven't done that yet, I'll do it in a later PR.

[tobiasschweizer]

I haven't done that yet, I'll do it in a later PR.

Ah, sorry, I was mistaken then.

[tobiasschweizer]

I didn't know that such a script existed.

We had a stall problem in Knora when doing the import and so @subotic added a load test script written in Python.

[subotic]

The load test can be found here: https://github.com/dhlab-basel/knora-tests

It is the import-test.

[subotic]

LGTM, thanks :-)

[subotic]

I like the upgrade scripts :-)

[benjamingeer]

I like the upgrade scripts :-)

Thanks, I was hoping you would like them. :)

[tobiasschweizer]

Could you please try updating the BEOL data using the scripts in upgrade/1230-delete-previews, and let me know if you find any problems?

I can confirm that upgrade/1230-delete-previews/delete-preview-values.py works.

I checked as follows:

1. I loaded beol-data.ttl with preview file values on develop
2. I ran this SPAQRL query:

PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
select * 
FROM <http://www.knora.org/data/0801/beol>
where { 
	    
    ?file a knora-base:StillImageFileValue .
    
    FILTER NOT EXISTS {
    	?file knora-base:isPreview true .
    } 
    
} ORDER BY ?file

3. I downloaded the result as CSV: ordered list of file value IRIs

4. I switched to wip/1068-remove-previews

5. I ran the script delete-preview-values.py

6. I ran this SPARQL query:

PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
select * 
FROM <http://www.knora.org/data/0801/beol>
where { 
	    
    ?file a knora-base:StillImageFileValue .
    
} ORDER BY ?file

6. I downloaded the result as CSV: ordered list of file value IRIs

7. I did a diff on the two CSV: no differences shown

#Questions:

• How would someone convert the data from an existing dump if this branch was already merged back to develop? The data would not load because of the consistency checking. Could there be a load script which would not check for consistency just for this case? Then the data could be written to a dump and reloaded with consistency checking.

• In Knora-ui, there is still some logic that deals with marking preview file values when reading them. This would not be necessary anymore. Could you please check with the Knora-ui team if this logic could be removed in a future version of Knora-ui? I think the current Knora-ui should work fine with this PR, the preview marking logic is just not necessary.

[benjamingeer]

I did a diff on the two CSV: no differences show

Great, thank you!

How would someone convert the data from an existing dump if this branch was already merged back to develop?

There would be no problem as long as their dump contains the old knora-base, which it would if it's a full backup of the repository.

[tobiasschweizer]

@kilchenmann is on holiday

[benjamingeer]

@kilchenmann @flavens Tobias says:

In Knora-ui, there is still some logic that deals with marking preview file values when reading them. This would not be necessary anymore. Could you please check with the Knora-ui team if this logic could be removed in a future version of Knora-ui? I think the current Knora-ui should work fine with this PR, the preview marking logic is just not necessary.

[tobiasschweizer]

@benjamingeer I will redo my Meditationes-import again this afternoon (converted data as basis). If problems occur, I would come back to you. Burt I think it should be fine.

[benjamingeer]

@tobiasschweizer OK, thanks very much.

[tobiasschweizer]

this is the issue in Knora-ui: dasch-swiss/knora-ui#190