Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add licenseUri to metadata records. #461

Merged
merged 5 commits into from
Apr 3, 2024
Merged

Add licenseUri to metadata records. #461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7faba66
Introduce licenseUri as additional field for Metadata(Schema)Record.
VolkerHartmann Feb 13, 2024
aed46d0
Add tests for licenseUri.
VolkerHartmann Feb 13, 2024
179e2af
Implement code for licenseUri.
VolkerHartmann Feb 13, 2024
31b3ea8
Add licenseUri as optional field for records to documentation.
VolkerHartmann Feb 13, 2024
b2a70b7
Improve code coverage.
VolkerHartmann Apr 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/documentation.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ like this:
"sid" : "...",
"permission" : "..."
} ],
"licenseUri" : "...",
"schemaDocumentUri" : "...",
"schemaHash" : "...",
"locked" : false
Expand All @@ -81,6 +82,8 @@ At least the following elements are expected to be provided by the user:
In addition, ACL may be useful to make schema readable/editable by others.
This will be of interest while accessing/updating an existing schema.(if authorization is enabled)

[NOTE]
License URI is optional. It's new since 1.5.0.

=== Registering a Metadata Schema Document

Expand Down Expand Up @@ -464,6 +467,7 @@ like this:
"sid": "...",
"permission": "..."
}],
"licenseUri": "...",
"metadataDocumentUri": "...",
"documentHash": "..."
}
Expand All @@ -479,6 +483,9 @@ In addition, ACL may be useful to make metadata editable by others. (This will b
[NOTE]
If linked schema is identified by its schemaId the INTERNAL type has to be used.

[NOTE]
License URI is optional. It's new since 1.5.0.

==== Register/Ingest a Metadata Record with Metadata Document

The following example shows the creation of the first metadata record and its metadata only providing mandatory fields mentioned above:
Expand Down Expand Up @@ -757,6 +764,7 @@ like this:
"sid" : "...",
"permission" : "..."
} ],
"licenseUri" : "...",
"schemaDocumentUri" : "...",
"schemaHash" : "...",
"locked" : false
Expand All @@ -771,6 +779,9 @@ At least the following elements are expected to be provided by the user:

In addition, ACL may be useful to make schema editable by others. (This will be of interest while updating an existing schema)

[NOTE]
License URI is optional. It's new since 1.5.0.

=== Registering a Metadata Schema Document

The following example shows the creation of the first json schema only providing mandatory fields mentioned above:
Expand Down Expand Up @@ -1186,6 +1197,7 @@ like this:
"sid": "...",
"permission": "..."
}],
"licenseUri": "...",
"metadataDocumentUri": "...",
"documentHash": "..."
}
Expand All @@ -1198,6 +1210,9 @@ At least the following elements are expected to be provided by the user:

In addition, ACL may be useful to make metadata editable by others. (This will be of interest while updating an existing metadata)

[NOTE]
License URI is optional. It's new since 1.5.0.

==== Register/Ingest a Metadata Record with Metadata Document

The following example shows the creation of the first metadata record and its metadata only providing mandatory fields mentioned above:
Expand Down
2 changes: 2 additions & 0 deletions src/main/java/edu/kit/datamanager/metastore2/domain/MetadataRecord.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ public class MetadataRecord implements EtagSupport, Serializable {
@NotNull(message = "A list of access control entries for resticting access.")
@OneToMany(cascade = jakarta.persistence.CascadeType.ALL, orphanRemoval = true)
private final Set<AclEntry> acl = new HashSet<>();
@NotBlank(message = "The uri of the license, e.g. for Apache-2.0 license this would be 'https://spdx.org/licenses/Apache-2.0'.")
private String licenseUri;
@NotBlank(message = "The metadata document uri, e.g. pointing to a local file.")
private String metadataDocumentUri;
@NotBlank(message = "The SHA-1 hash of the associated metadata file. The hash is used for comparison while updating.")
Expand Down
2 changes: 2 additions & 0 deletions src/main/java/edu/kit/datamanager/metastore2/domain/MetadataSchemaRecord.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ public enum SCHEMA_TYPE {
private Instant lastUpdate;
@Transient
private final Set<AclEntry> acl = new HashSet<>();
@NotBlank(message = "The uri of the license, e.g. for Apache-2.0 license this would be 'https://spdx.org/licenses/Apache-2.0'.")
private String licenseUri;
@NotBlank(message = "The schema document uri, e.g. pointing to a local file.")
private String schemaDocumentUri;
@NotBlank(message = "The SHA-1 hash of the associated schema file. The hash is used for comparison while synchonization.")
Expand Down
39 changes: 35 additions & 4 deletions src/main/java/edu/kit/datamanager/metastore2/util/MetadataRecordUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
import edu.kit.datamanager.repo.domain.Date;
import edu.kit.datamanager.repo.domain.RelatedIdentifier;
import edu.kit.datamanager.repo.domain.ResourceType;
import edu.kit.datamanager.repo.domain.Scheme;
import edu.kit.datamanager.repo.domain.Title;
import edu.kit.datamanager.repo.domain.acl.AclEntry;
import edu.kit.datamanager.repo.service.IContentInformationService;
Expand Down Expand Up @@ -445,6 +446,7 @@ public static DataResource migrateToDataResource(RepoBaseConfiguration applicati
dataResource.getTitles().add(Title.factoryTitle(defaultTitle, Title.TYPE.OTHER));
}
dataResource.setResourceType(ResourceType.createResourceType(MetadataRecord.RESOURCE_TYPE));
checkLicense(dataResource, metadataRecord.getLicenseUri());

return dataResource;
}
Expand Down Expand Up @@ -562,6 +564,10 @@ public static MetadataRecord migrateToMetadataRecord(RepoBaseConfiguration appli
}
}
}
// Only one license allowed. So don't worry about size of set.
if (!dataResource.getRights().isEmpty()) {
metadataRecord.setLicenseUri(dataResource.getRights().iterator().next().getSchemeUri());
}
long nano5 = System.nanoTime() / 1000000;
LOG.info("Migrate to MetadataRecord, {}, {}, {}, {}, {}, {}", nano1, nano2 - nano1, nano3 - nano1, nano4 - nano1, nano5 - nano1, provideETag);
}
Expand Down Expand Up @@ -622,7 +628,8 @@ public static MetadataSchemaRecord getCurrentInternalSchemaRecord(MetastoreConfi
URI finalUri = builder.build().toUri();

try {
returnValue = SimpleServiceClient.create(finalUri.toString()).withBearerToken(guestToken).accept(MetadataSchemaRecord.METADATA_SCHEMA_RECORD_MEDIA_TYPE).getResource(MetadataSchemaRecord.class);
returnValue = SimpleServiceClient.create(finalUri.toString()).withBearerToken(guestToken).accept(MetadataSchemaRecord.METADATA_SCHEMA_RECORD_MEDIA_TYPE).getResource(MetadataSchemaRecord.class
);
success = true;
break;
} catch (HttpClientErrorException ce) {
Expand Down Expand Up @@ -672,7 +679,8 @@ public static MetadataSchemaRecord getInternalSchemaRecord(MetastoreConfiguratio
URI finalUri = builder.build().toUri();

try {
returnValue = SimpleServiceClient.create(finalUri.toString()).withBearerToken(guestToken).accept(MetadataSchemaRecord.METADATA_SCHEMA_RECORD_MEDIA_TYPE).getResource(MetadataSchemaRecord.class);
returnValue = SimpleServiceClient.create(finalUri.toString()).withBearerToken(guestToken).accept(MetadataSchemaRecord.METADATA_SCHEMA_RECORD_MEDIA_TYPE).getResource(MetadataSchemaRecord.class
);
success = true;
break;
} catch (HttpClientErrorException ce) {
Expand Down Expand Up @@ -856,9 +864,10 @@ public static MetadataRecord mergeRecords(MetadataRecord managed, MetadataRecord
managed.setRelatedResource(mergeEntry("Updating record->relatedResource", managed.getRelatedResource(), provided.getRelatedResource()));
//update schemaId
managed.setSchema(mergeEntry("Updating record->schema", managed.getSchema(), provided.getSchema()));

//update schemaVersion
managed.setSchemaVersion(mergeEntry("Updating record->schemaVersion", managed.getSchemaVersion(), provided.getSchemaVersion()));
// update licenseUri
managed.setLicenseUri(mergeEntry("Updating record->licenseUri", managed.getLicenseUri(), provided.getLicenseUri(), true));
} else {
managed = (managed != null) ? managed : provided;
}
Expand Down Expand Up @@ -1048,7 +1057,29 @@ public static boolean checkAccessRights(Set<AclEntry> aclEntries, boolean curren

public static final void fixMetadataDocumentUri(MetadataRecord metadataRecord) {
String metadataDocumentUri = metadataRecord.getMetadataDocumentUri();
metadataRecord.setMetadataDocumentUri(WebMvcLinkBuilder.linkTo(WebMvcLinkBuilder.methodOn(MetadataControllerImpl.class).getMetadataDocumentById(metadataRecord.getId(), metadataRecord.getRecordVersion(), null, null)).toUri().toString());
metadataRecord
.setMetadataDocumentUri(WebMvcLinkBuilder.linkTo(WebMvcLinkBuilder.methodOn(MetadataControllerImpl.class
).getMetadataDocumentById(metadataRecord.getId(), metadataRecord.getRecordVersion(), null, null)).toUri().toString());
LOG.trace("Fix metadata document Uri '{}' -> '{}'", metadataDocumentUri, metadataRecord.getMetadataDocumentUri());
}

public static void checkLicense(DataResource dataResource, String licenseUri) {
if (licenseUri != null) {
Set<Scheme> rights = dataResource.getRights();
String licenseId = licenseUri.substring(licenseUri.lastIndexOf("/"));
Scheme license = Scheme.factoryScheme(licenseId, licenseUri);
if (rights.isEmpty()) {
rights.add(license);
} else {
// Check if license already exists (only one license allowed)
if (!rights.contains(license)) {
rights.clear();
rights.add(license);
}
}
} else {
// Remove license
dataResource.getRights().clear();
}
}
}
60 changes: 34 additions & 26 deletions src/main/java/edu/kit/datamanager/metastore2/util/MetadataSchemaRecordUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
import edu.kit.datamanager.repo.domain.Date;
import edu.kit.datamanager.repo.domain.Description;
import edu.kit.datamanager.repo.domain.ResourceType;
import edu.kit.datamanager.repo.domain.Scheme;
import edu.kit.datamanager.repo.domain.Title;
import edu.kit.datamanager.repo.service.IContentInformationService;
import edu.kit.datamanager.repo.util.ContentDataUtils;
Expand Down Expand Up @@ -444,6 +445,7 @@
checkDescription(descriptions, metadataSchemaRecord.getLabel(), Description.TYPE.OTHER);
checkDescription(descriptions, metadataSchemaRecord.getDefinition(), Description.TYPE.TECHNICAL_INFO);
checkDescription(descriptions, metadataSchemaRecord.getComment(), Description.TYPE.ABSTRACT);
MetadataRecordUtil.checkLicense(dataResource, metadataSchemaRecord.getLicenseUri());

return dataResource;
}
Expand Down Expand Up @@ -601,30 +603,34 @@
saveNewSchemaRecord(metadataSchemaRecord);
}
}
}
long nano7 = System.nanoTime() / 1000000;
// label -> description of type (OTHER)
// description -> description of type (TECHNICAL_INFO)
// comment -> description of type (ABSTRACT)
Iterator<Description> iterator = dataResource.getDescriptions().iterator();
while (iterator.hasNext()) {
Description nextDescription = iterator.next();
switch (nextDescription.getType()) {
case ABSTRACT:
metadataSchemaRecord.setComment(nextDescription.getDescription());
break;
case TECHNICAL_INFO:
metadataSchemaRecord.setDefinition(nextDescription.getDescription());
break;
case OTHER:
metadataSchemaRecord.setLabel(nextDescription.getDescription());
break;
default:
LOG.trace("Unknown description type: '{}' -> skipped", nextDescription.getType());
long nano7 = System.nanoTime() / 1000000;
// label -> description of type (OTHER)
// description -> description of type (TECHNICAL_INFO)
// comment -> description of type (ABSTRACT)
Iterator<Description> desc_iterator = dataResource.getDescriptions().iterator();
while (desc_iterator.hasNext()) {
Description nextDescription = desc_iterator.next();
switch (nextDescription.getType()) {
case ABSTRACT:
metadataSchemaRecord.setComment(nextDescription.getDescription());
break;
case TECHNICAL_INFO:
metadataSchemaRecord.setDefinition(nextDescription.getDescription());
break;
case OTHER:
metadataSchemaRecord.setLabel(nextDescription.getDescription());
break;
default:
LOG.trace("Unknown description type: '{}' -> skipped", nextDescription.getType());
}
}
// Only one license allowed. So don't worry about size of set.
if (!dataResource.getRights().isEmpty()) {
metadataSchemaRecord.setLicenseUri(dataResource.getRights().iterator().next().getSchemeUri());
}
if (LOG.isTraceEnabled()) {
LOG.trace("Migrate to schema record, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}", nano1, nano2 - nano1, nano3 - nano1, nano4 - nano1, nano4 - nano1, nano5 - nano1, nano6 - nano1, nano7 - nano1, provideETag);

Check warning on line 632 in src/main/java/edu/kit/datamanager/metastore2/util/MetadataSchemaRecordUtil.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/edu/kit/datamanager/metastore2/util/MetadataSchemaRecordUtil.java#L632 

Added line #L632 was not covered by tests
}
}
if (LOG.isTraceEnabled()) {
LOG.trace("Migrate to schema record, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}", nano1, nano2 - nano1, nano3 - nano1, nano4 - nano1, nano4 - nano1, nano5 - nano1, nano6 - nano1, nano7 - nano1, provideETag);
}
return metadataSchemaRecord;
}
Expand Down Expand Up @@ -916,9 +922,9 @@
MetadataSchemaRecord result = null;
Page<DataResource> dataResource;
try {
dataResource = metastoreProperties.getDataResourceService().findAllVersions(recordId, null);
dataResource = metastoreProperties.getDataResourceService().findAllVersions(recordId, null);
} catch (ResourceNotFoundException rnfe) {
rnfe.setDetail("Schema document with ID '" + recordId + "' doesn't exist!");
rnfe.setDetail("Schema document with ID '" + recordId + "' doesn't exist!");
throw rnfe;
}
long nano2 = System.nanoTime() / 1000000;
Expand All @@ -930,7 +936,7 @@
if (findFirst.isPresent()) {
result = migrateToMetadataSchemaRecord(metastoreProperties, findFirst.get(), supportEtag);
} else {
String message = String.format("Version '%d' of ID '%s' doesn't exist!",version, recordId);
String message = String.format("Version '%d' of ID '%s' doesn't exist!", version, recordId);
LOG.error(message);
throw new ResourceNotFoundException(message);
}
Expand Down Expand Up @@ -968,6 +974,8 @@
managed.setSchemaId(mergeEntry("Updating record->schema", managed.getSchemaId(), provided.getSchemaId()));
//update schemaVersion
managed.setSchemaVersion(mergeEntry("Updating record->schemaVersion", managed.getSchemaVersion(), provided.getSchemaVersion()));
// update licenseUri
managed.setLicenseUri(mergeEntry("Updating record->licenseUri", managed.getLicenseUri(), provided.getLicenseUri(), true));
} else {
managed = (managed != null) ? managed : provided;
}
Expand Down