Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IQSS/9387 - Support protected system metadata #9388

Merged
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
7f2c191
initial setting
qqmyers Jan 9, 2023
935f50c
UI cache for system md block setting
qqmyers Jan 9, 2023
549d813
Don't show system md blocks in edit.
qqmyers Jan 9, 2023
a4f1f1a
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Jan 17, 2023
bd49930
Bug fix in semantic api add method
qqmyers Jan 18, 2023
4cccef3
more bugs re: semantic methods
qqmyers Jan 18, 2023
0b3770e
Revert "more bugs re: semantic methods"
qqmyers Jan 18, 2023
ef13bcd
Revert "Bug fix in semantic api add method"
qqmyers Jan 18, 2023
ef233e2
@Inject works, @EJB doesn't
qqmyers Jan 19, 2023
bb7cf31
Adding comment
qqmyers Jan 19, 2023
a8981f3
add commented-out entry to microprofile properties
qqmyers Jan 19, 2023
3f795af
methods to identify changed md blocks
qqmyers Jan 19, 2023
4422940
implementation of system md block checks in Update/Create dsv commands
qqmyers Jan 19, 2023
3e36562
correcting/simplifying sem api methods
qqmyers Jan 19, 2023
0a89bbd
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Jan 19, 2023
156e070
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Jan 30, 2023
eafaa77
remove always false flag, add harvested flag
qqmyers Jan 30, 2023
e368c8a
Check for system md in create, except for harvested
qqmyers Jan 30, 2023
11cea54
add MetadataBlock to test DatasetFieldType
qqmyers Jan 30, 2023
c8680ab
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Feb 14, 2023
a36ab78
Fix checkstyle issue
qqmyers Feb 14, 2023
101580f
respond to review comments
qqmyers Feb 23, 2023
1f54785
allow headers or q params for system mdb keys
qqmyers Feb 23, 2023
dbc81fc
docs and release notes
qqmyers Feb 23, 2023
7dea429
fix docs
qqmyers Feb 23, 2023
33c4102
fix microprofile key parameterization
qqmyers Feb 23, 2023
3e03643
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Feb 23, 2023
f21cefb
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Mar 14, 2023
efeb049
Change to fine logging
qqmyers Mar 14, 2023
9b3d9a1
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Mar 22, 2023
a03263d
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Mar 28, 2023
2221d33
Merge remote-tracking branch 'IQSS/develop' into
qqmyers Apr 24, 2023
1c4228a
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Jul 5, 2023
e400f97
merge issue
qqmyers Jul 5, 2023
791b5cb
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Jul 10, 2023
5e1693d
don't show system blocks for edit in templates
qqmyers Jul 10, 2023
67d0618
fix create dataset constructor re: harvested/validation
qqmyers Jul 10, 2023
07fca52
Clarify case sensitive name, update example to use codeMeta20 name
qqmyers Jul 10, 2023
a3fe128
Merge remote-tracking branch 'IQSS/develop' into DANS/system_metadata
qqmyers Jul 10, 2023
cbfe78c
fix template create hiding of system blocks
qqmyers Jul 11, 2023
9ec4300
change examples to use codeVersion
qqmyers Jul 11, 2023
54f9f3f
typo - header had = not :
qqmyers Jul 11, 2023
c960a36
also mention template issue
qqmyers Jul 11, 2023
4130dfd
fix example (use specific schema.org mapping for this term)
qqmyers Jul 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/release-notes/9387-system_metadata_blocks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Dataverse supports requiring a secret key to add or edit metadata in specified 'system' metadata blocks. Changing the metadata in such system metadata blocks is not allowed without the key and is currently only allowed via API.
40 changes: 40 additions & 0 deletions doc/sphinx-guides/source/admin/metadatacustomization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,46 @@ The scripts required can be hosted locally or retrieved dynamically from https:/

Please note that in addition to the :ref:`:CVocConf` described above, an alternative is the :ref:`:ControlledVocabularyCustomJavaScript` setting.

Protecting MetadataBlocks
-------------------------

Dataverse can be configured to only allow entries for a metadata block to be changed (created, edited, deleted) by entities that know a defined secret key.
Metadata blocks protected by such a key are referred to as "System" metadata blocks.
A primary use case for system metadata blocks is to handle metadata created by third-party tools interacting with Dataverse where unintended changes to the metadata could cause a failure. Examples might include archiving systems or workflow engines.
To protect an existing metadatablock, one must set a key (recommended to be long and un-guessable) for that block:

dataverse.metadata.block-system-metadata-keys.<block name>=<key value>

This can be done using system properties (see :ref:`jvm-options`), environment variables or other MicroProfile Config mechanisms supported by the app server.
`See Payara docs for supported sources <https://docs.payara.fish/community/docs/documentation/microprofile/config/README.html#config-sources>`_.

For these secret keys, a password alias the "dir config source" of Payara are recommended.

Alias creation example using the codemeta metadata block:

.. code-block:: shell

echo "AS_ADMIN_ALIASPASSWORD=1234ChangeMeToSomethingLong" > /tmp/key.txt
asadmin create-password-alias --passwordfile /tmp/key.txt dataverse.metadata.block-system-metadata-keys.codemeta
rm /tmp/key.txt

When protected via a key, a metadata block will not be shown in the user interface when a dataset is being created or when metadata is being edited. Entries in such a system metadata block will be shown to users, consistent with Dataverse's design in which all metadata in published datasets is publicly visible.

To add metadata to a system metadata block via API, one must include an additional key of the form

mdkey.<blockName>=<key value>

as an HTTP Header or query parameter for each system metadata block to any API call in which metadata values are changed in that block. Multiple keys are allowed if more than one system metadatablock is being changed in a given API call.

For example, following the :ref:`Add Dataset Metadata <add-semantic-metadata>` example from the :doc:`/developers/dataset-semantic-metadata-api`:

.. code-block:: bash

curl -X PUT -H X-Dataverse-key:$API_TOKEN -H 'Content-Type: application/ld+json' -H 'mdkey.codemeta=1234' -d '{"title": "Submit menu test", "@context":{"title": "http://purl.org/dc/terms/title"}}' "$SERVER_URL/api/datasets/$DATASET_ID/metadata

curl -X PUT -H X-Dataverse-key:$API_TOKEN -H 'Content-Type: application/ld+json' -d '{"title": "Submit menu test", "@context":{"title": "http://purl.org/dc/terms/title"}}' "$SERVER_URL/api/datasets/$DATASET_ID/metadata?mdkey.codemeta=1234


Tips from the Dataverse Community
---------------------------------

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ To get the json-ld formatted metadata for a Dataset, specify the Dataset ID (DAT
You should expect a 200 ("OK") response and JSON-LD mirroring the OAI-ORE representation in the returned 'data' object.


.. _add-semantic-metadata:

Add Dataset Metadata
--------------------

Expand Down
2 changes: 1 addition & 1 deletion src/main/java/edu/harvard/iq/dataverse/DatasetPage.java
Original file line number Diff line number Diff line change
Expand Up @@ -3595,7 +3595,7 @@ public String save() {
//ToDo - could drop use of selectedTemplate and just use the persistent dataset.getTemplate()
if ( selectedTemplate != null ) {
if ( isSessionUserAuthenticated() ) {
cmd = new CreateNewDatasetCommand(dataset, dvRequestService.getDataverseRequest(), false, selectedTemplate);
cmd = new CreateNewDatasetCommand(dataset, dvRequestService.getDataverseRequest(), selectedTemplate);
} else {
JH.addMessage(FacesMessage.SEVERITY_FATAL, BundleUtil.getStringFromBundle("dataset.create.authenticatedUsersOnly"));
return null;
Expand Down
114 changes: 110 additions & 4 deletions src/main/java/edu/harvard/iq/dataverse/DatasetVersionDifference.java
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,29 @@

import edu.harvard.iq.dataverse.datavariable.DataVariable;
import edu.harvard.iq.dataverse.datavariable.VarGroup;
import edu.harvard.iq.dataverse.datavariable.VariableMetadata;
import edu.harvard.iq.dataverse.datavariable.VariableMetadataUtil;
import edu.harvard.iq.dataverse.util.StringUtil;

import java.util.ArrayList;
import java.util.Collections;
import java.util.Collection;
import java.util.List;
import java.util.Set;
import java.util.logging.Logger;

import org.apache.commons.lang3.StringUtils;
import edu.harvard.iq.dataverse.util.BundleUtil;
import edu.harvard.iq.dataverse.util.FileUtil;

import java.util.Arrays;
import java.util.Date;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedList;

/**
*
* @author skraffmiller
*/
public final class DatasetVersionDifference {
private static final Logger logger = Logger.getLogger(DatasetVersionDifference.class.getCanonicalName());

private DatasetVersion newVersion;
private DatasetVersion originalVersion;
Expand Down Expand Up @@ -1713,4 +1714,109 @@ public void setDatasetFilesDiffList(List<datasetFileDifferenceItem> datasetFiles
this.datasetFilesDiffList = datasetFilesDiffList;
}

/*
* Static methods to compute which blocks have changes between the two
* DatasetVersions. Currently used to assess whether 'system metadatablocks'
* (protected by a separate key) have changed. (Simplified from the methods
* above that track all the individual changes)
*
*/
public static Set<MetadataBlock> getBlocksWithChanges(DatasetVersion newVersion, DatasetVersion originalVersion) {
Set<MetadataBlock> changedBlockSet = new HashSet<MetadataBlock>();

// Compare Data
List<DatasetField> newDatasetFields = new LinkedList<DatasetField>(newVersion.getDatasetFields());
if (originalVersion == null) {
// Every field is new, just list blocks used
Iterator<DatasetField> dsfnIter = newDatasetFields.listIterator();
while (dsfnIter.hasNext()) {
DatasetField dsfn = dsfnIter.next();
if (!changedBlockSet.contains(dsfn.getDatasetFieldType().getMetadataBlock())) {
changedBlockSet.add(dsfn.getDatasetFieldType().getMetadataBlock());
}
}

} else {
List<DatasetField> originalDatasetFields = new LinkedList<DatasetField>(originalVersion.getDatasetFields());
Iterator<DatasetField> dsfoIter = originalDatasetFields.listIterator();
while (dsfoIter.hasNext()) {
DatasetField dsfo = dsfoIter.next();
boolean deleted = true;
Iterator<DatasetField> dsfnIter = newDatasetFields.listIterator();

while (dsfnIter.hasNext()) {
DatasetField dsfn = dsfnIter.next();
if (dsfo.getDatasetFieldType().equals(dsfn.getDatasetFieldType())) {
deleted = false;
if (!changedBlockSet.contains(dsfo.getDatasetFieldType().getMetadataBlock())) {
logger.fine("Checking " + dsfo.getDatasetFieldType().getName());
if (dsfo.getDatasetFieldType().isPrimitive()) {
if (fieldsAreDifferent(dsfo, dsfn, false)) {
logger.fine("Adding block for " + dsfo.getDatasetFieldType().getName());
changedBlockSet.add(dsfo.getDatasetFieldType().getMetadataBlock());
}
} else {
if (fieldsAreDifferent(dsfo, dsfn, true)) {
logger.fine("Adding block for " + dsfo.getDatasetFieldType().getName());
changedBlockSet.add(dsfo.getDatasetFieldType().getMetadataBlock());
}
}
}
dsfnIter.remove();
break; // if found go to next dataset field
}
}

if (deleted) {
logger.fine("Adding block for deleted " + dsfo.getDatasetFieldType().getName());
changedBlockSet.add(dsfo.getDatasetFieldType().getMetadataBlock());
}
dsfoIter.remove();
}
// Only fields left are non-matching ones but they may be empty
for (DatasetField dsfn : newDatasetFields) {
if (!dsfn.isEmpty()) {
logger.fine("Adding block for added " + dsfn.getDatasetFieldType().getName());
changedBlockSet.add(dsfn.getDatasetFieldType().getMetadataBlock());
}
}
}
return changedBlockSet;
}

private static boolean fieldsAreDifferent(DatasetField originalField, DatasetField newField, boolean compound) {
String originalValue = "";
String newValue = "";

if (compound) {
for (DatasetFieldCompoundValue datasetFieldCompoundValueOriginal : originalField
.getDatasetFieldCompoundValues()) {
int loopIndex = 0;
if (newField.getDatasetFieldCompoundValues().size() >= loopIndex + 1) {
for (DatasetField dsfo : datasetFieldCompoundValueOriginal.getChildDatasetFields()) {
if (!dsfo.getDisplayValue().isEmpty()) {
originalValue += dsfo.getDisplayValue() + ", ";
}
}
for (DatasetField dsfn : newField.getDatasetFieldCompoundValues().get(loopIndex)
.getChildDatasetFields()) {
if (!dsfn.getDisplayValue().isEmpty()) {
newValue += dsfn.getDisplayValue() + ", ";
}
}
if (!originalValue.trim().equals(newValue.trim())) {
return true;
}
}
loopIndex++;
}
} else {
originalValue = originalField.getDisplayValue();
newValue = newField.getDisplayValue();
if (!originalValue.equalsIgnoreCase(newValue)) {
return true;
}
}
return false;
}
}
17 changes: 9 additions & 8 deletions src/main/java/edu/harvard/iq/dataverse/DatasetVersionUI.java
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,18 @@
package edu.harvard.iq.dataverse;

import edu.harvard.iq.dataverse.util.MarkupChecker;
import edu.harvard.iq.dataverse.util.StringUtil;
import java.io.Serializable;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import static java.util.stream.Collectors.toList;
import javax.ejb.EJB;
import javax.faces.view.ViewScoped;
import javax.inject.Named;
import javax.inject.Inject;
import javax.json.JsonObject;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;

Expand All @@ -35,6 +30,9 @@ public class DatasetVersionUI implements Serializable {

@EJB
DataverseServiceBean dataverseService;
@Inject
SettingsWrapper settingsWrapper;

@PersistenceContext(unitName = "VDCNet-ejbPU")
private EntityManager em;

Expand Down Expand Up @@ -400,6 +398,9 @@ public void setMetadataValueBlocks(DatasetVersion datasetVersion) {
//TODO: A lot of clean up on the logic of this method
metadataBlocksForView.clear();
metadataBlocksForEdit.clear();

List<MetadataBlock> systemMDBlocks = settingsWrapper.getSystemMetadataBlocks();

Long dvIdForInputLevel = datasetVersion.getDataset().getOwner().getId();

if (!dataverseService.find(dvIdForInputLevel).isMetadataBlockRoot()){
Expand Down Expand Up @@ -442,7 +443,7 @@ public void setMetadataValueBlocks(DatasetVersion datasetVersion) {
if (!datasetFieldsForView.isEmpty()) {
metadataBlocksForView.put(mdb, datasetFieldsForView);
}
if (!datasetFieldsForEdit.isEmpty()) {
if (!datasetFieldsForEdit.isEmpty() && !systemMDBlocks.contains(mdb)) {
metadataBlocksForEdit.put(mdb, datasetFieldsForEdit);
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@ public class MetadataBlockServiceBean {
@PersistenceContext(unitName = "VDCNet-ejbPU")
private EntityManager em;



public MetadataBlock save(MetadataBlock mdb) {
return em.merge(mdb);
}
Expand Down
23 changes: 23 additions & 0 deletions src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
package edu.harvard.iq.dataverse;

import edu.harvard.iq.dataverse.branding.BrandingUtil;
import edu.harvard.iq.dataverse.settings.JvmSettings;
import edu.harvard.iq.dataverse.settings.Setting;
import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
import edu.harvard.iq.dataverse.settings.SettingsServiceBean.Key;
Expand Down Expand Up @@ -35,6 +36,7 @@
import javax.faces.validator.ValidatorException;
import javax.faces.view.ViewScoped;
import javax.inject.Named;
import javax.json.Json;
import javax.json.JsonObject;
import javax.mail.internet.InternetAddress;

Expand All @@ -59,6 +61,9 @@ public class SettingsWrapper implements java.io.Serializable {

@EJB
DatasetFieldServiceBean fieldService;

@EJB
MetadataBlockServiceBean mdbService;

private Map<String, String> settingsMap;

Expand Down Expand Up @@ -115,6 +120,8 @@ public class SettingsWrapper implements java.io.Serializable {

private Boolean customLicenseAllowed = null;

private List<MetadataBlock> systemMetadataBlocks;

private Set<Type> alwaysMuted = null;

private Set<Type> neverMuted = null;
Expand Down Expand Up @@ -700,4 +707,20 @@ public boolean isCustomLicenseAllowed() {
}
return customLicenseAllowed;
}

public List<MetadataBlock> getSystemMetadataBlocks() {

if (systemMetadataBlocks == null) {
systemMetadataBlocks = new ArrayList<MetadataBlock>();
}
List<MetadataBlock> blocks = mdbService.listMetadataBlocks();
for (MetadataBlock mdb : blocks) {
String smdbString = JvmSettings.MDB_SYSTEM_KEY_FOR.lookupOptional(mdb.getName()).orElse(null);
if (smdbString != null) {
systemMetadataBlocks.add(mdb);
}
}

return systemMetadataBlocks;
}
}