500 server error when adding a dataset in dataverse where default template has no associated terms

[scolapasta]

This was reported by a user via RT:
They click Add Dataset on their dataverse and get a 500 error page.

The reason is that a null pointer is being thrown. After some investigation, it turns out that the dataset has a default template with no associated terms (termsofuseandaccess_id is null on the template) and that in turn causes the null pointer on line 163 of DatasetUtil:
License license = dsv.getTermsOfUseAndAccess().getLicense();

The workaround we suggested is to go and edit the template and associate terms (even if empty) to the template.

I did try to reproduce on demo with a new dataverse, but I wasn't able to create a template with no terms (I tried briefly) so it's possible that can't be done anymore and only affects some legacy templates (in prod, there were 89 such templates, 48 of which were set as a default template).

The fix could be one of:
• db update to create empty terms for any existing templates (and verify there is no way to create any such new ones)
• change line 163 to make sure dsv.getTermsOfUseAndAccess() is not null before calling getLicense()

[scolapasta]

Looks like we may need to opt for some aspect of 1 or more complicated 2) as right now you also cannot edit the terms on these legacy templates (an error is thrown).

[scolapasta]

So, to fix the issue for the one that was from the RT issue, I:
• Created a new dummy template (through ui)
• through db set the associated terms of the existing template to the ones created for the dummy template, and set the dummy template terms to null.
• deleted dummy template (through ui)

Note that means the template now has a cc0 license, so this would need to be reviewed to make sure that is what is desired. If not, the dataverse admin can then change it through the UI.

[qqmyers]

From some quick testing at QDR, I don't think you can create new templates that don't have a termsofuseandaccess so this might be a job for flyway. FWIW: I played with this commit to start - basically null check TermsOfUseAndAccess 'everywhere' it is being used to get the license (everywhere doesn't include in the xhtml files). Without a flyway, this would cause users of an old template to get custom terms instead of the default license though.

[mreekie]

Priority - the current work around is a manual fix. We are getting RT tickets on this issue so this is a production impacting issue. It will likely impact other dataverse implementations outside of harvard.edu

[mreekie]

sprint

• writeup a db script
• release to community.
• small

[scolapasta]

This query can be used to fix individual templates. Eventually, will see if possible to handle problematic templates in bulk.

Note this sets the fileaccessrequest in the template to false and the license_id to 1. We should verify those values (but we need to choose something for both, and that is the default if you just create a new template from the UI).

with _insert as 
(
insert into termsofuseandaccess (fileaccessrequest, license_id) values (false, 1)
returning id 
)
update template set termsofuseandaccess_id =  (select id from _insert)
where id = XXX and termsofuseandaccess_id is null;

[scolapasta]

Foxing multiple in one query is a little tricky, but this query at least can be used multiple times (the last one was id specific) - this one will grab the first one without a terms, create a new terms, and links them. i.e. if you run a 2nd time, it will then grab the next one without terms...

with _sel as 
(
select id from template where termsofuseandaccess_id is null limit 1
), _insert as 
(
insert into termsofuseandaccess (fileaccessrequest, license_id) values (false, 1)
returning id 
)
update template set termsofuseandaccess_id =  (select id from _insert)
from _sel where template.id=_sel.id;

[scolapasta]

Ah, actually the way to do it is in reverse - rather than first insert and then update (the problem being that the update then needs to take from a stack), you first update, then insert. This uses the "stacking" inherent in nextval.

(I thought you might have to disable, then reenable the foreign key, but it seems to work without that, probably because it's all handled in the one query).

with  _update as 
(
update template set termsofuseandaccess_id =  nextval('termsofuseandaccess_id_seq' )
where termsofuseandaccess_id is null
returning termsofuseandaccess_id
)
insert into termsofuseandaccess (id, fileaccessrequest, license_id) (select termsofuseandaccess_id, false, 1 from _update)

[mreekie]

Sprint

• 8599: Still small; I just need to find time to make the flyway script / release notes

[mreekie]

• in WIP at end of pm.sprint.2022_05_11

[mreekie]

Sprint:

• pm.sprint.2022_05_25 ended WIP