5565 allow edit of file position in directory structure

doc/release-notes/5565-file-folder-names.md

@@ -0,0 +1,3 @@
+In this release users are given an option to edit the folder names in the file metadata. Strict validation rules for the folder names are also introduced. Only the following characters are allowed: the alphanumerics, '_', '-', '.' and ' ' (white space). Some datafiles in your Dataverse may already have folder names saved in the database (if they were extracted from uploaded zip archives with folder structure). The following sanitizing rules will be applied to all the existing folder names in the database: any invalid characters will be replaced by the '.' character. Any sequences of dots will be further replaced with a single dot. For example, the folder name ``data&info/code=@137`` will be converted to ``data.info/code.137``. This update will be automatically applied to the database the first time this release is deployed.
+
+

doc/sphinx-guides/source/api/native-api.rst

@@ -440,7 +440,7 @@ Add a file to an existing Dataset. Description and tags are optional::
 
 A more detailed "add" example using curl::
 
-    curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' -F 'jsonData={"description":"My description.","categories":["Data"], "restrict":"true"}' "https://example.dataverse.edu/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"
+    curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' -F 'jsonData={"description":"My description.","directoryLabel":"data/subdir1","categories":["Data"], "restrict":"true"}' "https://example.dataverse.edu/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"
 
 Example python code to add a file. This may be run by changing these parameters in the sample code:
 
@@ -719,12 +719,12 @@ A curl example using a ``pid``::
 Replacing Files
 ~~~~~~~~~~~~~~~
 
-Replace an existing file where ``id`` is the database id of the file to replace or ``pid`` is the persistent id (DOI or Handle) of the file. Note that metadata such as description and tags are not carried over from the file being replaced
+Replace an existing file where ``id`` is the database id of the file to replace or ``pid`` is the persistent id (DOI or Handle) of the file. Note that metadata such as description, directoryLabel (File Path) and tags are not carried over from the file being replaced
 
 .. code-block:: bash
 
   curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F 'file=@data.tsv' \
-  -F 'jsonData={"description":"My description.","categories":["Data"],"forceReplace":false}'\
+  -F 'jsonData={"description":"My description.","directoryLabel":"data/subdir1","categories":["Data"],"forceReplace":false}'\
   "https://demo.dataverse.org/api/files/$FILE_ID/replace"
 
 Example python code to replace a file.  This may be run by changing these parameters in the sample code:

doc/sphinx-guides/source/developers/big-data-support.rst

@@ -103,6 +103,7 @@ Optional steps for setting up the S3 Docker DCM Variant
 
   - Add AWS bucket info to dcmsrv
     - Add AWS credentials to ``~/.aws/credentials``
+
       - ``[default]``
       - ``aws_access_key_id =``
       - ``aws_secret_access_key =``
@@ -117,6 +118,7 @@ Optional steps for setting up the S3 Docker DCM Variant
 
   - Add AWS bucket info to Dataverse
     - Add AWS credentials to ``~/.aws/credentials``
+
       - ``[default]``
       - ``aws_access_key_id =``
       - ``aws_secret_access_key =``

doc/sphinx-guides/source/developers/sql-upgrade-scripts.rst

@@ -26,7 +26,7 @@ How to Create a SQL Upgrade Script
 
 We assume you have already read the :doc:`version-control` section and have been keeping your feature branch up to date with the "develop" branch.
 
-Create a new file called something like ``V4.11__5513-database-variablemetadata.sql`` in the ``src/main/resources/db/migration`` directory. Use the previously released version (4.11 in the example above) rather than what we guess will be the next released version (which is often uncertain). For the "description" you should the name of your branch, which should include the GitHub issue you are working on, as in the example above. To read more about Flyway file naming conventions, see https://flywaydb.org/documentation/migrations#naming
+Create a new file called something like ``V4.11.0.1__5565-sanitize-directory-labels.sql`` in the ``src/main/resources/db/migration`` directory. Use a version like "4.11.0.1" in the example above where the previously released version was 4.11, ensuring that the version number is unique.  For the "description" you should the name of your branch, which should include the GitHub issue you are working on, as in the example above. To read more about Flyway file naming conventions, see https://flywaydb.org/documentation/migrations#naming
 
 The SQL migration script you wrote will be part of the war file and executed when the war file is deployed. To see a history of Flyway database migrations that have been applied, look at the ``flyway_schema_history`` table.
 

doc/sphinx-guides/source/installation/config.rst

@@ -1108,6 +1108,8 @@ If you don't want to register file-based PIDs for your installation, set:
 
 Note: File-level PID registration was added in 4.9 and is required until version 4.9.3.
 
+.. _:IndependentHandleService:
+
 :IndependentHandleService
 +++++++++++++++++++++++++++
 

doc/sphinx-guides/source/user/dataset-management.rst

@@ -180,11 +180,15 @@ Metadata found in the header section of `Flexible Image Transport System (FITS)
 Compressed Files
 ----------------
 
-Compressed files in zip format are unpacked automatically. If it fails to unpack, for whatever reason, it will upload as is. If the number of files inside are more than a set limit (1,000 by default, configurable by the Administrator), you will get an error message and the zip file will uploads as is.
+Compressed files in .zip format are unpacked automatically. If a .zip file fails to unpack for whatever reason, it will upload as is. If the number of files inside are more than a set limit (1,000 by default, configurable by the Administrator), you will get an error message and the .zip file will upload as is.
+
+If the uploaded .zip file contains a folder structure, Dataverse will keep track of this structure. A file's location within this folder structure is displayed in the file metadata as the File Path. When you download the contents of the dataset, this folder structure will be preserved and files will appear in their original locations. 
+
+These folder names are subject to strict validation rules. Only the following characters are allowed: the alphanumerics, '_', '-', '.' and ' ' (white space). When a zip archive is uploaded, the folder names are automatically sanitized, with any invalid characters replaced by the '.' character. Any sequences of dots are further replaced with a single dot. For example, the folder name ``data&info/code=@137`` will be converted to ``data.info/code.137``. When uploading through the Web UI, the user can change the values further on the edit form presented, before clicking the 'Save' button. 
+
+.. note:: If you upload multiple .zip files to one dataset, any subdirectories that are identical across multiple .zips will be merged together when the user downloads the full dataset.
 
-.. note:: If the uploaded zip file contains sub-folders, the names of the folders will be preserved as the ``DirectoryLabel`` attributes in the ``FileMetadata`` objects of the corresponding individual datafiles. As of writing this - v.4.11 - these labels are only used to restore the folder structure in multi-file, zipped download bundles (see :doc:`/api/dataaccess` for more information). In the future folder structure will be supported for organizing files on the dataset page as well. 
 
-Support for unpacking tar files will be added when this ticket is closed: https://github.com/IQSS/dataverse/issues/2195.
 
 Other File Types
 ----------------
@@ -212,6 +216,14 @@ You will not have to leave the dataset page to complete these action, except for
 
 If you restrict files, you will also prompted with a popup asking you to fill out the Terms of Access for the files. If Terms of Access already exist, you will be asked to confirm them. Note that some Dataverse installations do not allow for file restrictions.
 
+File Path
+---------
+
+The File Path metadata field is Dataverse's way of representing a file's location in a folder structure. When a user uploads a .zip file containing a folder structure, Dataverse automatically fills in the File Path information for each file contained in the .zip. If a user downloads the full dataset or a selection of files from it, they will receive a folder structure with each file positioned according to its File Path.
+
+A file's File Path can be manually added or edited on the Edit Files page. Changing a file's File Path will change its location in the folder structure that is created when a user downloads the full dataset or a selection of files from it.
+
+
 File Tags
 ---------
 

doc/sphinx-guides/source/user/find-use-data.rst

@@ -74,7 +74,7 @@ You can find the citation for the dataset at the top of the dataset page in a bl
 Download Files
 --------------
 
-Within the Files tab on a dataset page, you can download the files in that dataset. To download more than one file at a time, select the files you would like to download and then click the Download button above the files. The selected files will download in zip format.
+Within the Files tab on a dataset page, you can download the files in that dataset. To download more than one file at a time, select the files you would like to download and then click the Download button above the files. The selected files will download in .zip format that preserves any folder structure that the dataset owner had set up.
 
 You may also download a file from its file page by clicking the Download button in the upper right corner of the page, or by :ref:`url_download` under the Metadata tab on the lower half of the page.
 

src/main/java/Bundle.properties

@@ -1728,6 +1728,8 @@ file.metadataTab.fileMetadata.type.label=Type
 file.metadataTab.fileMetadata.description.label=Description
 file.metadataTab.fileMetadata.publicationDate.label=Publication Date
 file.metadataTab.fileMetadata.depositDate.label=Deposit Date
+file.metadataTab.fileMetadata.hierarchy.label=File Path
+file.metadataTab.fileMetadata.hierarchy.tip=Hierarchical directory structure path used to display file organization and support reproducibility.
 file.metadataTab.fitsMetadata.header=FITS Metadata
 
 file.versionDifferences.noChanges=No changes associated with this version

src/main/java/ValidationMessages.properties

@@ -30,7 +30,7 @@ alias.illegalCharacters=Alias cannot be empty. Valid characters are a-Z, 0-9, '_
 custom.response=Please enter a response.
 custom.questiontext=Please enter question text.
 filename.illegalCharacters=File Name cannot contain any of the following characters: \ / : * ? " < > | ; # .
-directoryname.illegalCharacters=Directory Name cannot contain leading or trailing file separators.
+directoryname.illegalCharacters=Directory Name cannot contain invalid characters. Valid characters are a-Z, 0-9, '_', '-', '.', '\\', '/' and ' ' (white space).
 filename.blank=Please specify a file name.
 
 

src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java

@@ -69,13 +69,19 @@
 import org.apache.commons.httpclient.methods.GetMethod;
 import java.text.DateFormat;
 import java.util.Arrays;
+import java.util.HashSet;
 import java.util.ResourceBundle;
 import java.util.Set;
 import java.util.logging.Level;
 import javax.faces.event.AjaxBehaviorEvent;
 import javax.faces.event.FacesEvent;
+import javax.faces.event.ValueChangeEvent;
 import javax.servlet.ServletOutputStream;
 import javax.servlet.http.HttpServletResponse;
+import javax.validation.ConstraintViolation;
+import javax.validation.Validation;
+import javax.validation.Validator;
+import javax.validation.ValidatorFactory;
 import org.apache.commons.lang.StringUtils;
 import org.primefaces.context.RequestContext;
 
@@ -1060,26 +1066,6 @@ public String saveReplacementFile() throws FileReplaceException{
 
     public String save() {
 
-
-        /*
-        // Validate
-        Set<ConstraintViolation> constraintViolations = workingVersion.validate();
-        if (!constraintViolations.isEmpty()) {
-             //JsfHelper.addFlashMessage(getBundleString("dataset.message.validationError"));
-            logger.fine("Constraint violation detected on SAVE: "+constraintViolations.toString());
-             JH.addMessage(FacesMessage.SEVERITY_ERROR, getBundleString("dataset.message.validationError"));
-
-            //FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Validation Error", "See below for details."));
-            return "";
-        }
-        }*/
-
-        // Once all the filemetadatas pass the validation, we'll only allow the user 
-        // to try to save once; (this it to prevent them from creating multiple
-        // DRAFT versions, if the page gets stuck in that state where it 
-        // successfully creates a new version, but can't complete the remaining
-        // tasks. -- L.A. 4.2
-
         if (!saveEnabled) {
             return "";
         }
@@ -2760,6 +2746,10 @@ public void handleDescriptionChange(final AjaxBehaviorEvent event) {
     public void handleNameChange(final AjaxBehaviorEvent event) {        
         datasetUpdateRequired = true;
     }
+
+    public void handleFileDirectoryChange(final ValueChangeEvent event) {  
+        datasetUpdateRequired = true;
+    }
 
     /* 
      * Items for the "Advanced (Ingest) Options" popup. 

src/main/java/edu/harvard/iq/dataverse/FileDirectoryNameValidator.java

@@ -0,0 +1,43 @@
+/*
+ * To change this license header, choose License Headers in Project Properties.
+ * To change this template file, choose Tools | Templates
+ * and open the template in the editor.
+ */
+package edu.harvard.iq.dataverse;
+
+import edu.harvard.iq.dataverse.util.StringUtil;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import javax.validation.ConstraintValidator;
+import javax.validation.ConstraintValidatorContext;
+
+/**
+ *
+ * @author skraffmi
+ */
+public class FileDirectoryNameValidator implements ConstraintValidator<ValidateDataFileDirectoryName, String> {
+
+    @Override
+    public void initialize(ValidateDataFileDirectoryName constraintAnnotation) {
+
+    }
+
+    @Override
+    public boolean isValid(String value, ConstraintValidatorContext context) {
+        return isFileDirectoryNameValid(value, context);
+
+    }
+
+    public static boolean isFileDirectoryNameValid(String value, ConstraintValidatorContext context) {
+
+        if (value == null || value.isEmpty()) {
+            return true;
+        }
+        String validCharacters = "[\\w\\\\/. -]+";
+        Pattern p = Pattern.compile(validCharacters);
+        Matcher m = p.matcher(value);
+        return m.matches();
+
+    }
+
+}

src/main/java/edu/harvard/iq/dataverse/FileMetadata.java

@@ -33,6 +33,7 @@
 import javax.persistence.Version;
 
 import edu.harvard.iq.dataverse.util.DateUtil;
+import edu.harvard.iq.dataverse.util.StringUtil;
 import org.hibernate.validator.constraints.NotBlank;
 import javax.validation.constraints.Pattern;
 
@@ -55,10 +56,11 @@ public class FileMetadata implements Serializable {
     @Column( nullable=false )
     private String label = "";
 
-    @Pattern(regexp="|[^/\\\\]|^[^/\\\\]+.*[^/\\\\]+$",
-            message = "{directoryname.illegalCharacters}")
+
+    @ValidateDataFileDirectoryName(message = "{directoryname.illegalCharacters}")
     @Expose
     @Column ( nullable=true )
+
     private String directoryLabel;
     @Column(columnDefinition = "TEXT")
     private String description = "";
@@ -120,6 +122,12 @@ public String getDirectoryLabel() {
     }
 
     public void setDirectoryLabel(String directoryLabel) {
+        //Strip off beginning and ending \ // - .
+        // and replace any sequences/combinations of / and \ with a single /
+        if (directoryLabel != null) {
+            directoryLabel = StringUtil.sanitizeFileDirectory(directoryLabel);
+        }
+
         this.directoryLabel = directoryLabel;
     }
 

src/main/java/edu/harvard/iq/dataverse/ValidateDataFileDirectoryName.java

@@ -0,0 +1,32 @@
+/*
+ * To change this license header, choose License Headers in Project Properties.
+ * To change this template file, choose Tools | Templates
+ * and open the template in the editor.
+ */
+package edu.harvard.iq.dataverse;
+
+import java.lang.annotation.Documented;
+import static java.lang.annotation.ElementType.FIELD;
+import java.lang.annotation.Retention;
+import static java.lang.annotation.RetentionPolicy.RUNTIME;
+import java.lang.annotation.Target;
+import javax.validation.Constraint;
+import javax.validation.Payload;
+
+/**
+ *
+ * @author skraffmi
+ */
+@Target({FIELD})
+@Retention(RUNTIME)
+@Constraint(validatedBy = {FileDirectoryNameValidator.class})
+@Documented
+public @interface ValidateDataFileDirectoryName {
+
+    String message() default "Failed Validation for Validate Data File Directory Name";
+
+    Class<?>[] groups() default {};
+
+    Class<? extends Payload>[] payload() default {};
+
+}

src/main/java/edu/harvard/iq/dataverse/datasetutility/OptionalFileParams.java

@@ -42,6 +42,9 @@ public class OptionalFileParams {
     private String description;
     public static final String DESCRIPTION_ATTR_NAME = "description";
 
+    private String directoryLabel;
+    public static final String DIRECTORY_LABEL_ATTR_NAME = "directoryLabel";
+
     private List<String> categories;
     public static final String CATEGORIES_ATTR_NAME = "categories";
 
@@ -99,7 +102,15 @@ public void setDescription(String description){
     public String getDescription(){
         return this.description;
     }
-
+
+    public String getDirectoryLabel() {
+        return directoryLabel;
+    }
+
+    public void setDirectoryLabel(String directoryLabel) {
+        this.directoryLabel = directoryLabel;
+    }
+
     public void setRestriction(boolean restrict){
         this.restrict = restrict;
     }
@@ -128,6 +139,13 @@ public boolean hasDescription(){
         }
         return true;
     }
+
+    public boolean hasDirectoryLabel(){
+        if ((directoryLabel == null)||(this.directoryLabel.isEmpty())){
+            return false;
+        }
+        return true;
+    }
 
     /**
      *  Set tags
@@ -192,7 +210,15 @@ private void loadParamsFromJson(String jsonData) throws DataFileTagException{
 
             this.description = jsonObj.get(DESCRIPTION_ATTR_NAME).getAsString();
         }
-
+
+        // -------------------------------
+        // get directory label as string
+        // -------------------------------
+        if ((jsonObj.has(DIRECTORY_LABEL_ATTR_NAME)) && (!jsonObj.get(DIRECTORY_LABEL_ATTR_NAME).isJsonNull())){
+
+            this.directoryLabel = jsonObj.get(DIRECTORY_LABEL_ATTR_NAME).getAsString();
+        }
+
         // -------------------------------
         // get restriction as boolean
         // -------------------------------
@@ -295,8 +321,14 @@ public void addOptionalParams(DataFile df) throws DataFileTagException{
         if (hasDescription()){
             fm.setDescription(this.getDescription());
         }
-
-
+
+        // ---------------------------
+        // Add directory label (path)
+        // ---------------------------
+        if (hasDirectoryLabel()){
+            fm.setDirectoryLabel(this.getDirectoryLabel());
+        }
+
         // ---------------------------
         // Add categories
         // ---------------------------