Extend Datasets getDownloadSize API endpoint to support file search criteria and deaccessioned datasets

[GPortas]

What this PR does / why we need it:

Extended the getDownloadSize endpoint (/api/datasets/{id}/versions/{versionId}/downloadsize), including the following new features:

• The endpoint now accepts a new boolean optional query parameter "includeDeaccessioned", which, if enabled, causes the endpoint to consider deaccessioned dataset versions when searching for versions to obtain the file total download size.

• The endpoint now supports filtering by criteria. In particular, it accepts the following optional criteria query parameters:

  • contentType
  • accessStatus
  • categoryName
  • tabularTagName
  • searchText

Which issue(s) this PR closes:

Closes #9995

Special notes for your reviewer:

This extension is required for the SPA to display deaccessioned dataset versions.

Suggestions on how to test this:

Create a dataset.

Upload files to the dataset and test the endpoint via curl.

This is a call which already worked before this PR. For ignoring the tabular original sizes:

curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "http://localhost:8080/api/datasets/24/versions/1.0/downloadsize?mode=Archival"

Add one of the new criteria options, for example "contentType".

curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "http://localhost:8080/api/datasets/24/versions/1.0/downloadsize?mode=Archival&contentType=image/png"

Here we are using contentType=image/png. Replace the value with the kind of content type you are looking for.

Deaccession the dataset and ensure you can obtain the same information as before when sending an API key linked to a user with permissions and specifying includeDeaccessioned=true.

curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "http://localhost:8080/api/datasets/24/versions/1.0/downloadsize?mode=Archival&contentType=image/png&includeDeaccessioned=true"

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No

Is there a release notes update needed for this change?:

Yes

Additional documentation:

N/A

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:9995-file-total-download-size-criteria

ghcr.io/gdcc/configbaker:9995-file-total-download-size-criteria

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

[sekmiller]

Does it make sense to list some or all of the possible content types here?

[GPortas]

There are many different content types (https://github.com/IQSS/dataverse/blob/develop/src/main/java/propertyFiles/MimeTypeFacets.properties).

Being something so extensive and variable, I'm not sure if it would be really useful.

[sekmiller]

Looks good overall. Just a question about adding more to the doc and a reminder to get the latest from dev

[kcondon]

@GPortas @don I'm seeing build error on Jenkins:
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] SiteMapUtilTest.testUpdateSiteMap:107 Unexpected exception thrown: java.io.FileNotFoundException: /tmp/junit4920710667909896148/docroot/sitemap/sitemap.xml (No such file or directory)
[INFO]
[ERROR] Tests run: 1542, Failures: 1, Errors: 0, Skipped: 32
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.555 s
[INFO] Finished at: 2023-10-18T17:25:12-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.0:test (default-test) on project dataverse: There are test failures.
[ERROR]
[ERROR] Please refer to /home/worker/workspace/IQSS_Dataverse_Internal/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
[PostBuildScript] - [INFO] Executing post build scripts.
[PostBuildScript] - [INFO] Build does not have any of the results [SUCCESS]. Did not execute build step #0.
[PostBuildScript] - [INFO] Build does not have any of the results [SUCCESS]. Did not execute build step #1.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: FAILURE

[GPortas]

@kcondon @donsizemore

I don't see any errors in actions or my localhost. It seems more like a problem with how the directories are configured in Jenkins (?) Since it tries to access the file in the /tmp directory and cannot find it.

Also, the component under test is not related to the changes in this PR.

[kcondon]

@GPortas @donsizemore It's building now, thanks.