directoryLabel leading '.' char for hidden directories is ignored (removed)

[jernsting]

What steps does it take to reproduce the issue?
Crate a new file in a dataset with directoryLabel starting with '.' char. And save the file.

• When does this issue occur?
  Issue occurs when adding files in hidden directories on top level. Hidden directories inside other directories work.

• Which page(s) does it occurs on?
  Upload dataverse files (API and webinterface)

• What happens?
  The leading '.' char is ignored / removed. For example adding .hidden/file will be saved as hidden/file.

• To whom does it occur (all users, curators, superusers)?
  all users

• What did you expect to happen?
  Adding .hidden/file should be saved in a hidden directory named .hidden

Which version of Dataverse are you using?
v. 5.9 build 750-fb24c87

Any related open or closed issues to this bug report?
datalad/datalad-dataverse#83

Screenshots:

• Adding file in hidden dir
  
• hidden dir is now no longer hidden
  

[bpoldrack]

For the record: Same thing appears to happen at demo.dataverse.org with slightly newer dataverse.

[poikilotherm]

Hat tip to @qqmyers for pointing me in the right direction.

The problem is in FileMetadata.setDirectoryLabel() -> StringUtil.sanitizeFileDirectory():

dataverse/src/main/java/edu/harvard/iq/dataverse/util/StringUtil.java

Lines 177 to 179 in 4686066

	while (value.startsWith("/") || value.startsWith("-") || value.startsWith(".") || value.startsWith(" ")){
	value = value.substring(1);
	}

@landreev @sekmiller according to git history, you wrote the code - there is no statement about why you cut off a leading dot, but this has obviously been a design decision. Before creating a PR to exclude single starting dots, maybe you can shed some light on this (I know, code is from 2019...)? Thx!

There is a parameterized test that also demonstrates the experienced behavior is as designed.

dataverse/src/test/java/edu/harvard/iq/dataverse/util/StringUtilTest.java

Line 252 in 4686066

{".some/path/to/a/directory", "some/path/to/a/directory", false},

)

[pdurbin]

Here's the release note from 4.12: https://github.com/IQSS/dataverse/releases/tag/v4.12

---

A note on folder names:

In this release users are given an option to edit the folder names in the file metadata. Strict validation rules for the folder names are also introduced. Only the following characters are allowed: the alphanumerics, '_', '-', '.' and ' ' (white space). Some datafiles in your Dataverse may already have folder names saved in the database (if they were extracted from uploaded zip archives with folder structure). The following sanitizing rules will be applied to all the existing folder names in the database: any invalid characters will be replaced by the '.' character. Any sequences of dots will be further replaced with a single dot. For example, the folder name data&info/code=@137 will be converted to data.info/code.137. This update will be automatically applied to the database the first time this release is deployed.

---

From the above it seems like period/dot/full stop/. should be allowed. That note was added in PR #5650 which closed #5565.

As observed in this issue it seems like leading periods are disallowed. This is reflected not only in the code above but in the SQL update script in the pull request above:

-- get rid of any leading or trailing slashes, spaces, '-'s and '.'s: 
UPDATE filemetadata SET directoryLabel = regexp_replace(directoryLabel, '^[/ .\-]+', '', '');

[pdurbin]

If we supported dot directories like .datalad it would be easier for DataLad folks to find their datasets that are hosted on Dataverse:

• Support discovery of datalad datasets on dataverse datalad/datalad-usage-dashboard#46

Besides, it would be generally useful as well, of course! 😄

Reading my comment above, I'm wondering if dot directories are already supported. 🤔 We should re-test.

[landreev]

@pdurbin
Similarly, I cannot remember why we chose to implement it like that, stripping the initial dot... yet it definitely looks like we did so on purpose. 🤔 indeed.
Interestingly, it is only the initial dot that is stripped. I.e., if you enter .hidden it is auto-converted to hidden. But subfolder/.hidden is left intact.
We should probably just go ahead and stop stripping it - ?

[pdurbin]

@landreev it sounds like you looked at the code. Great. Yes, we should stop removing the initial .. Interesting that they're allowed in subfolders!