Collection Quotas (8549)

[landreev]

(working on adding the "how to test" section now)

What this PR does / why we need it:

Which issue(s) this PR closes:

Closes #8549

Special notes for your reviewer:
I'd like to double-check/have a second and third opinions etc. on the viability of this approach as described below.

[another edit: In line with the scope of the issue, the quotas mechanism added is for collections only. But nothing in the implementation prevents from extending it to datasets as well.]

The main idea behind the implementation is to maintain a real-time record of storage use for all the DvObjectContainers (Datasets and Collections), since we cannot possibly afford to calculate it on the fly for every file upload. We record storage use for the entire hierarchy of nested objects. I.e., adding a file to the dataset increments the storage use for the parent dataset, and all the parent collections up to the root.

This information is stored in its own dedicated table, StorageUse. As opposed to storing it as an extra column in the DvObject table, since that would require updating all the parent DvObjects every time we upload a file, likely causing merge conflicts on a busy instance/large collections.

These updates are performed via native recursive queries in new transactions, avoiding any cascading updates.

These recursive updates need to be performed not just every time a new file is uploaded, but every time a file is successfully ingested (this adds the size of the archival tab-delimited file to the total), when an unpublished file is deleted and when a tabular file is un-ingested.

I'm only counting the sizes of the main datafiles, and the produced tab-delimited versions for the ingested tabular files for these purposes. I.e., I consider these to be the archival "payload" of datasets and collections, as opposed to all the file- and dataset-level auxiliary files, such as the resized image thumbnails and metadata exports, under the assumption that those are transitive, i.e., can be deleted and automatically re-generated. If necessary, we can extend the scheme to also count the sizes of (some, select?) auxiliaries going forward. [edit: In the context of the IQSS prod. instance, this feature is specifically needed for the larger collections and datasets, on the order of TBs or similar, where the sizes of these aux. files should not amount to any statistically significant portion of the overall storage]

I chose to have this system of keeping the real-time record of storage use automatically enabled, regardless of whether quota enforcement is configured on the instance. With the rationale being that this information can be very useful regardless. (The existing commands for calculating storage sizes perform this in real time and are thus less efficient). Also, the idea is that these updates via direct native queries should be fairly efficient and not result in any serious overhead. A somewhat bulky flyway script is provided to populate this map for the existing objects. I tested it on a copy of the prod. db and the performance of the current version is adequate. But, like with everything else in the PR, I would appreciate another developer taking a quick look at it.

Suggestions on how to test this:

The functionality is mostly for the admins (both instance admins, aka "superusers", who can set, change and delete these quotas for specific collection, and collection admins as in, users who own and manage collections, who will have read access and be able to see the quotas defined on their collections and the storage use already accumulated on them). It is for the most part API-only. I.e., at this point we don't want to invest much work into adding anything to the existing UI.

The only UI impact is that the end user of a collection that has a quota configured will be warned about it on the file upload page, as in

... and of course they will be getting error messages on that page attempting to upload anything that goes over the quota.

So the way to test the functionality would be to

• create a collection
• use a superuser api token to configure a quota, along the lines of
  curl -X POST -H "X-Dataverse-key: xxx" http://localhost:8080/api/dataverses/<COLLECTIONALIAS>/storage/quota/1048576
  (probably makes sense to use something stingy like that for testing...)
• use the settings api to enable quota enforcement on the instance: (configuring a quota on a specific collection does not by itself enable the enforcement!)
  curl -X PUT -d true http://localhost:8080/api/admin/settings/:UseStorageQuotas
• proceed to upload files, make sure files can be uploaded until the specified byte size is reached, and that Dataverse starts rejecting further uploads once that happens...

So the above is the simplest functionality test. How much effort to invest into trying to find a way to break the feature is a judgement call/matter of balance between thoroughness and efficiency.

Reviewing the documentation introduced in the pr is an integral part of QA. The goal should be to try and read it from the point of view of an admin managing a Dataverse instance somewhere else - the intended audience of the guide - and see whether it's sufficient for them to be able to use the feature, if it's clear enough etc.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

[coveralls]

coverage: 19.983% (-0.02%) from 20.006%
when pulling cda06a3 on 8549-collection-quotas
into 1ea692b on develop.

[landreev]

This class was used during the phase one/proof of concept development.

[pdurbin]

First pass review. Overall, looks good. I hope performance is good enough! It'll be a great feature, one that's often asked for!

[pdurbin]

Now that we have a API changelog we could say something like, "See the API changelog for details: http://preview.guides.gdcc.io/en/develop/api/changelog.html "

[landreev]

Will do.

[pdurbin]

This makes me wonder if a quota can be defined globally or not. That is, a default quota for any random collection.

(Time passes.)

I see from some Javadoc below there is some inheritance going on. This should be added to the guides somewhere.

     * Checks if the supplied DvObjectContainer...
     * has a quota configured, and if not, keeps checking if any of the direct
     * ancestor Collections further up have a configured quota. If it finds one, 
     * it will retrieve the current total content size for that specific ancestor 
     * dvObjectContainer and use it to define the quota limit for the upload
     * session in progress. 

[landreev]

Yes, there's definitely going to be more documentation, and more tests. I'm still working on these parts, and this is the main reason this is still a draft PR. But I really wanted other developers to look at the mechanics of the implementation.

[scolapasta]

I think (correct me if I'm wrong - that that comment is less about a default global quota, and more that since collections can contain collection you have to go up the tree.

[pdurbin]

It shouldn't be too hard to add tests for direct upload and quotas once we merge this PR:

• add S3 tests, LocalStack, MinIO #10044

[pdurbin]

Suggested change

	/*
	* Click nbfs://nbhost/SystemFileSystem/Templates/Licenses/license-default.txt to change this license
	* Click nbfs://nbhost/SystemFileSystem/Templates/Classes/Class.java to edit this template
	*/

[pdurbin]

Yes, great to continue this testing.

[pdurbin]

Good that we're thinking about the "many files" case.

[landreev]

Yes, this is still one of the work-in-progress/still experimenting with parts. I may have a better solution in the works).

[pdurbin]

Oh good, tabs are being removed, I think, I hope.

[pdurbin]

Any UI impact? Should we add to the long, dynamic list of rules? Or defer until we're off JSF? Screenshot below:

[landreev]

Yes, the remaining quota info will be shown to the user on the file upload page, above. The code for that is already in the develop branch, it was added in the proof of concept/phase one pr (#9409).

[landreev]

Like this:

[landreev]

My one big concern about this scheme is its reliance on this new StorageUse table that needs to be modified every time a file (or a batch of files) are saved successfully. Specifically, that the one entry in it, that for the root collection, will need to be updated any time a file is uploaded anywhere else in the dvobject tree. The same applies to any large collection, or any collection where multiple active uploads are happening in parallel in different parts of the tree. The update itself is quite simple and fast, and the order in which the actual increments are applied is irrelevant, but I'm still wondering if there is some potential race condition that can lock things up (?).
In practical terms my response is that we'll never know for sure until we deploy the feature on a busy prod. instance like ours; and then, in case this ends up causing any database conflicts, I added a kill switch, a JVM/mpconfig option that stops the updates from happening in real time.

[stevenwinship]

I'm doing the QA now. What if anything should happen if the quota is set to a value less than the current size of the existing data?

[landreev]

I'm doing the QA now. What if anything should happen if the quota is set to a value less than the current size of the existing data?

The collection will become over the storage quota limit right away, thus disabling further file uploads.
So, this can be considered a way for an admin to make a specific collection read-only.

[github-actions]

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:8549-collection-quotas

ghcr.io/gdcc/configbaker:8549-collection-quotas

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.