7551 expanded compound datasetfield validation

doc/release-notes/7551-expanded-compound-datasetfield-validation.md

@@ -0,0 +1,36 @@
+## Notes for Dataverse Administrators 
+
+Prior to this release, when defining metadata for compound fields (via their dataset field types), fields could be either be optional or required, i.e. if required you must always have (at least one) value for that field. For example, Author Name being required means you must have at least one Author with an nonempty Author name.
+
+In order to support more robust metadata (and specifically to resolve #7551), we need to allow a third case: Conditionally Required, that is, the field is required if and only if any of its "sibling" fields are entered. For example, Producer Name is now conditionally required in the citation metadata block. A user does not have to enter a Producer, but if they do, they have to enter a Producer Name.
+
+This change required some modifications to how "required" is defined in the metadata .tsv files (for compound fields).
+
+Prior to this release, the value of required for the parent compound field did not matter and so was set to false.
+
+Going forward:
+
+- For optional, the parent compound field would be required = false and all children would be required = false.
+- For required, the parent compound field would be required = true and at least one child would be required = true.
+- For conditionally required, the parent compound field would be required = false and at least one child would be required = true.
+
+This release updates the citation .tsv file that is distributed with the software for the required parent compound fields (e.g. author), as well as sets Producer Name to be conditionally required. No other distributed .tsv files were updated, as they did not have any required compound values.
+
+**If you have created any custom metadata .tsv files**, you will need to make the same (type of) changes there.
+
+### Additional Upgrade Steps
+
+1. Reload Citation Metadata Block:
+
+   `wget https://github.com/IQSS/dataverse/releases/download/v5.4/citation.tsv`
+   `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`
+
+2. Update any custom metadata blocks (if used):
+
+   For any subfield that has a required value of TRUE, find the corresponding parent field and change its required value to TRUE.
+
+   Note: As there is an accompanying Flyway script that updates the values directly in the database, you do not need to reload these metadata .tsv files via API, unless you make additional changes, e.g set some compound fields to be conditionally required.
+
+### Use Case
+
+Metadata designers can now set subfields of compound fields as **conditionally required**, that is, the field is required if and only if any of its "sibling" fields are entered. For example, Producer Name is now conditionally required in the citation metadata block. A user does not have to enter a Producer, but if they do, they have to enter a Producer Name.

doc/sphinx-guides/source/admin/metadatacustomization.rst

@@ -9,7 +9,7 @@ The Dataverse Software has a flexible data-driven metadata system powered by "me
 Introduction
 ------------
 
-Before you embark on customizing metadata in your Dataverse installation you should make sure you are aware of the modest amount of customization that is available with your Dataverse installation's web interface. It's possible to hide fields and make field required by clicking "Edit" at the Dataverse collection level, clicking "General Information" and making adjustments under "Metadata Fields" as described in the :ref:`create-dataverse` section of the Dataverse Collection Management page in the User Guide.
+Before you embark on customizing metadata in your Dataverse installation you should make sure you are aware of the modest amount of customization that is available with your Dataverse installation's web interface. It's possible to hide fields and make fields required or conditionally required by clicking "Edit" at the Dataverse collection level, clicking "General Information" and making adjustments under "Metadata Fields" as described in the :ref:`create-dataverse` section of the Dataverse Collection Management page in the User Guide.
 
 Much more customization of metadata is possible, but this is an advanced topic so feedback on what is written below is very welcome. The possibilities for customization include:
 
@@ -289,16 +289,37 @@ Each of the three main sections own sets of properties:
 |                       | the dataset has been  |                        |
 |                       | saved.                |                        |
 +-----------------------+-----------------------+------------------------+
-| required              | Specify whether or    | TRUE (required) or     |
-|                       | not the field is      | FALSE (optional)       |
-|                       | required. This means  |                        |
-|                       | that at least one     |                        |
-|                       | instance of the field |                        |
-|                       | must be present. More |                        |
-|                       | than one field may be |                        |
-|                       | allowed, depending on |                        |
-|                       | the value of          |                        |
-|                       | allowmultiples.       |                        |
+| required              | For primitive         | For primitive          |
+|                       | fields, specify       | fields, TRUE           |
+|                       | whether or not the    | (required) or FALSE    |
+|                       | field is required.    | (optional).            |
+|                       | For compound          |                        |
+|                       | fields, also          | For compound fields:   |
+|                       | specify if one or     |                        |
+|                       | more subfields are    | \• To make one or more |
+|                       | required or           | subfields optional,    |
+|                       | conditionally         | the parent field and   |
+|                       | required. At least    | subfield(s) must be    |
+|                       | one instance of a     | FALSE (optional).      |
+|                       | required field must   |                        |
+|                       | be present. More      | \• To make one or more |
+|                       | than one instance     | subfields required,    |
+|                       | of a field may be     | the parent field and   |
+|                       | allowed, depending    | the required           |
+|                       | on the value of       | subfield(s) must be    |
+|                       | allowmultiples.       | TRUE (required).       |
+|                       |                       |                        |
+|                       |                       | \• To make one or more |
+|                       |                       | subfields              |
+|                       |                       | conditionally          |
+|                       |                       | required, make the     |
+|                       |                       | parent field FALSE     |
+|                       |                       | (optional) and make    |
+|                       |                       | TRUE (required) any    |
+|                       |                       | subfield or subfields  |
+|                       |                       | that are required if   |
+|                       |                       | any other subfields    |
+|                       |                       | are filled.            |
 +-----------------------+-----------------------+------------------------+
 | parent                | For subfields,        | \• Must not result in  |
 |                       | specify the name of   | a cyclical             |

doc/sphinx-guides/source/user/dataverse-management.rst

@@ -28,8 +28,8 @@ Creating a Dataverse collection is easy but first you must be a registered user
 #. **Choose the sets of Metadata Fields for datasets in this Dataverse collection**:
     * By default the metadata elements will be from the host Dataverse collection that this new Dataverse collection is created in.
     * The Dataverse Software offers metadata standards for multiple domains. To learn more about the metadata standards in the Dataverse Software please check out the :doc:`/user/appendix`.
-    * Metadata fields can be hidden, or selected as required or optional.
-    * Selected metadata elements are also used to pick which metadata fields you would like to use for creating :ref:`Dataset Templates <dataset-templates>`: after you finish creating your Dataverse collection.
+    * Most metadata fields can be hidden or can be selected as required or optional. Some fields may be selected as conditionally required, depending on metadata options chosen for the Dataverse installation.
+    * Selected metadata fields are also used to pick which metadata fields you would like to use for creating :ref:`Dataset Templates <dataset-templates>`: after you finish creating your Dataverse collection.
 #. **Choose which metadata fields will be used as browse/search facets on your Dataverse collection**:
     * These facets will allow users browsing or searching your Dataverse collection to filter its contents according to the fields you have selected. For example, if you select “Subject” as a facet, users will be able to filter your Dataverse collection’s contents by subject area.
     * By default, the facets that will appear on your Dataverse collection's landing page will be from the host Dataverse collection that this new Dataverse collection was created in, but you can add or remove facets from this default.

scripts/api/data/metadatablocks/citation.tsv

@@ -8,16 +8,16 @@
 	otherId	Other ID	Another unique identifier that identifies this Dataset (e.g., producer's or another repository's number).		none	4	:	FALSE	FALSE	TRUE	FALSE	FALSE	FALSE		citation
 	otherIdAgency	Agency	Name of agency which generated this identifier.		text	5	#VALUE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	otherId	citation
 	otherIdValue	Identifier	Other identifier that corresponds to this Dataset.		text	6	#VALUE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	otherId	citation
-	author	Author	The person(s), corporate body(ies), or agency(ies) responsible for creating the work.		none	7		FALSE	FALSE	TRUE	FALSE	TRUE	FALSE		citation	http://purl.org/dc/terms/creator
+	author	Author	The person(s), corporate body(ies), or agency(ies) responsible for creating the work.		none	7		FALSE	FALSE	TRUE	FALSE	TRUE	TRUE		citation	http://purl.org/dc/terms/creator
 	authorName	Name	The author's Family Name, Given Name or the name of the organization responsible for this Dataset.	FamilyName, GivenName or Organization	text	8	#VALUE	TRUE	FALSE	FALSE	TRUE	TRUE	TRUE	author	citation
 	authorAffiliation	Affiliation	The organization with which the author is affiliated.		text	9	(#VALUE)	TRUE	FALSE	FALSE	TRUE	TRUE	FALSE	author	citation
 	authorIdentifierScheme	Identifier Scheme	Name of the identifier scheme (ORCID, ISNI).		text	10	- #VALUE:	FALSE	TRUE	FALSE	FALSE	TRUE	FALSE	author	citation	http://purl.org/spar/datacite/AgentIdentifierScheme
 	authorIdentifier	Identifier	Uniquely identifies an individual author or organization, according to various schemes.		text	11	#VALUE	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	author	citation	http://purl.org/spar/datacite/AgentIdentifier
-	datasetContact	Contact	The contact(s) for this Dataset.		none	12		FALSE	FALSE	TRUE	FALSE	TRUE	FALSE		citation
+	datasetContact	Contact	The contact(s) for this Dataset.		none	12		FALSE	FALSE	TRUE	FALSE	TRUE	TRUE		citation
 	datasetContactName	Name	The contact's Family Name, Given Name or the name of the organization.	FamilyName, GivenName or Organization	text	13	#VALUE	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	datasetContact	citation
 	datasetContactAffiliation	Affiliation	The organization with which the contact is affiliated.		text	14	(#VALUE)	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	datasetContact	citation
 	datasetContactEmail	E-mail	The e-mail address(es) of the contact(s) for the Dataset. This will not be displayed.		email	15	#EMAIL	FALSE	FALSE	FALSE	FALSE	TRUE	TRUE	datasetContact	citation
-	dsDescription	Description	A summary describing the purpose, nature, and scope of the Dataset.		none	16		FALSE	FALSE	TRUE	FALSE	TRUE	FALSE		citation	
+	dsDescription	Description	A summary describing the purpose, nature, and scope of the Dataset.		none	16		FALSE	FALSE	TRUE	FALSE	TRUE	TRUE		citation	
 	dsDescriptionValue	Text	A summary describing the purpose, nature, and scope of the Dataset.		textbox	17	#VALUE	TRUE	FALSE	FALSE	FALSE	TRUE	TRUE	dsDescription	citation
 	dsDescriptionDate	Date	In cases where a Dataset contains more than one description (for example, one might be supplied by the data producer and another prepared by the data repository where the data are deposited), the date attribute is used to distinguish between the two descriptions. The date attribute follows the ISO convention of YYYY-MM-DD.	YYYY-MM-DD	date	18	(#VALUE)	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	dsDescription	citation
 	subject	Subject	Domain-specific Subject Categories that are topically relevant to the Dataset.		text	19		TRUE	TRUE	TRUE	TRUE	TRUE	TRUE		citation	http://purl.org/dc/terms/subject
@@ -37,7 +37,7 @@
 	notesText	Notes	Additional important information about the Dataset.		textbox	33		FALSE	FALSE	FALSE	FALSE	TRUE	FALSE		citation
 	language	Language	Language of the Dataset		text	34		TRUE	TRUE	TRUE	TRUE	FALSE	FALSE		citation	http://purl.org/dc/terms/language
 	producer	Producer	Person or organization with the financial or administrative responsibility over this Dataset		none	35		FALSE	FALSE	TRUE	FALSE	FALSE	FALSE		citation
-	producerName	Name	Producer name	FamilyName, GivenName or Organization	text	36	#VALUE	TRUE	FALSE	FALSE	TRUE	FALSE	FALSE	producer	citation
+	producerName	Name	Producer name	FamilyName, GivenName or Organization	text	36	#VALUE	TRUE	FALSE	FALSE	TRUE	FALSE	TRUE	producer	citation
 	producerAffiliation	Affiliation	The organization with which the producer is affiliated.		text	37	(#VALUE)	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	producer	citation
 	producerAbbreviation	Abbreviation	The abbreviation by which the producer is commonly known. (ex. IQSS, ICPSR)		text	38	(#VALUE)	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	producer	citation
 	producerURL	URL	Producer URL points to the producer's web presence, if appropriate. Enter an absolute URL where the producer's web site is found, such as http://www.my.org.  	Enter full URL, starting with http://	url	39	<a href="#VALUE" target="_blank" rel="noopener">#VALUE</a>	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	producer	citation

src/main/java/edu/harvard/iq/dataverse/DatasetField.java

@@ -407,52 +407,80 @@ public void setValidationMessage(String validationMessage) {
         this.validationMessage = validationMessage;
     }
 
+
+    // these two booleans are used to wrap around the field type isRequired and isHasChildren methods to also check
+    // if the dataverse has overriden the default values
     @Transient 
     private Boolean required;
+    @Transient 
+    private Boolean hasRequiredChildren;
 
     public boolean isRequired() {
         if (required == null) {
             required = false;
-            if (this.datasetFieldType.isPrimitive() && this.datasetFieldType.isRequired()) {
+
+            if (this.datasetFieldType.isRequired()) {
                 required = true;
-            }
-
-            if (this.datasetFieldType.isHasRequiredChildren()) {
-                required = true;
-            }
-
-            Dataverse dv = getDataverse();
-            while (!dv.isMetadataBlockRoot()) {
-                if (dv.getOwner() == null) {
-                    break; // we are at the root; which by defintion is metadata blcok root, regarldess of the value
+            } else {
+                // otherwise check if overridden at the dataverse level
+                Dataverse dv = getDataverse();
+                while (!dv.isMetadataBlockRoot()) {
+                    if (dv.getOwner() == null) {
+                        break; // we are at the root; which by defintion is metadata blcok root, regarldess of the value
+                    }
+                    dv = dv.getOwner();
                 }
-                dv = dv.getOwner();
-            }
 
-            List<DataverseFieldTypeInputLevel> dftilListFirst = dv.getDataverseFieldTypeInputLevels();
-            if (!getDatasetFieldType().isHasChildren()) {
+                List<DataverseFieldTypeInputLevel> dftilListFirst = dv.getDataverseFieldTypeInputLevels();
                 for (DataverseFieldTypeInputLevel dsftil : dftilListFirst) {
                     if (dsftil.getDatasetFieldType().equals(this.datasetFieldType)) {
                         required = dsftil.isRequired();
                     }
                 }
             }
-
-            if (getDatasetFieldType().isHasChildren() && (!dftilListFirst.isEmpty())) {
-                for (DatasetFieldType child : getDatasetFieldType().getChildDatasetFieldTypes()) {
-                    for (DataverseFieldTypeInputLevel dftilTest : dftilListFirst) {
-                        if (child.equals(dftilTest.getDatasetFieldType())) {
-                            if (dftilTest.isRequired()) {
-                                required = true;
-                            }
+
+            // since we don't currently support the use case where the parent is required
+            // but no specific children are (the "true/false" case), we override this as false
+            if (required && this.datasetFieldType.isCompound() && !isHasRequiredChildren())
+            {
+                required = false;
+            }
+        }
+
+        return required;
+    }
+
+    public boolean isHasRequiredChildren() {
+        if (hasRequiredChildren == null) {
+            hasRequiredChildren = false;
+        }
+
+        if (this.datasetFieldType.isHasRequiredChildren()) {
+            hasRequiredChildren = true;
+        }        
+
+        Dataverse dv = getDataverse();
+        while (!dv.isMetadataBlockRoot()) {
+            if (dv.getOwner() == null) {
+                break; // we are at the root; which by defintion is metadata blcok root, regarldess of the value
+            }
+            dv = dv.getOwner();
+        }        
+
+        List<DataverseFieldTypeInputLevel> dftilListFirst = dv.getDataverseFieldTypeInputLevels();
+
+        if (getDatasetFieldType().isHasChildren() && (!dftilListFirst.isEmpty())) {
+            for (DatasetFieldType child : getDatasetFieldType().getChildDatasetFieldTypes()) {
+                for (DataverseFieldTypeInputLevel dftilTest : dftilListFirst) {
+                    if (child.equals(dftilTest.getDatasetFieldType())) {
+                        if (dftilTest.isRequired()) {
+                            hasRequiredChildren = true;
                         }
                     }
                 }
             }
-
-        }
-        // logger.fine("at return  " + this.datasetFieldType.getDisplayName() + " " + required);
-        return required;
+        }        
+        return hasRequiredChildren;
     }
 
     public Dataverse getDataverse() {

src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java

@@ -141,7 +141,9 @@ public void setOptionSelectItems(List<SelectItem> optionSelectItems) {
     public DatasetFieldType() {}
 
     public DatasetFieldType(String name, FieldType fieldType, boolean allowMultiples) {
+        // use the name for both default name and title
         this.name = name;
+        this.title = name;
         this.fieldType = fieldType;
         this.allowMultiples = allowMultiples;
         childDatasetFieldTypes = new LinkedList<>();