Setting "General Terms of Use" longer than 1024 characters via API fails, resulting in "500 Internal Server Error"

[pdurbin]

If you try to set the "General Terms of Use" using a file with more than 1024 characters as described at http://guides.dataverse.org/en/4.2.1/installation/installation-main.html#applicationtermsofuse like @shlake mentioned on the mailing list, it will fail with the following in the stack trace:

"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(1024)
Error Code: 0
Call: INSERT INTO ACTIONLOGRECORD (ID, ACTIONRESULT, ACTIONSUBTYPE, ACTIONTYPE, ENDTIME, INFO, STARTTIME, USERIDENTIFIER) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
bind => [8 parameters bound]"

The reason for this is that writing a setting falls under "Log important events to an audit log" implemented in #1532 and the actionlogrecord table has a field called "info" that is limited to 1024 characters:

$ grep -C1 1024 src/main/java/edu/harvard/iq/dataverse/actionlogging/ActionLogRecord.java

    @Column(length = 1024)
    private String info;

Somehow we got more than 1024 characters in place at https://dataverse.harvard.edu but my guess is that it was a direct database write to the setting table to avoid writing to the actionlogrecord table. This is a potential workaround that should be documented if it helps.

I'm attaching a stack trace from a0bc225 which is the commit I was on when I reproduced the bug using the steps below. I'll also attach the file mentioned on the mailing list by @shlake that exercises the bug. (I'm appending ".txt" to the filename because GitHub doesn't support upload of HTML files.)

murphy:dataverse pdurbin$ wc -c /tmp/uva/Scratch2.html
    1082 /tmp/uva/Scratch2.html
murphy:dataverse pdurbin$ curl -s -i http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse -X PUT -d@/tmp/uva/Scratch2.html | head
HTTP/1.1 100 Continue

HTTP/1.1 500 Internal Server Error
Server: GlassFish Server Open Source Edition  4.1 
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition  4.1  Java/Oracle Corporation/1.8)
Date: Thu, 22 Oct 2015 15:42:56 GMT
Set-Cookie: JSESSIONID=037a3b557c70592a8c66a412280c; Path=/; HttpOnly
Content-Type: application/xhtml+xml;charset=UTF-8
Connection: close
Transfer-Encoding: chunked
(23) Failed writing body

• stacktrace-a0bc225.txt
• Scratch2.html.txt

[bencomp]

I see other possibilities than circumventing the action log:

• change the type of the info field to text ("clob") to fit the whole text
• limiting the size of the text that is put in the action log
• storing the actual terms text elsewhere and referring it in the action log record.

[pdurbin]

@bencomp yes, these are excellent ideas. Another possibility:

• do not persist settings changes to the actionlog

When I said workaround I meant something that @shlake could do today with the version she's running (4.2). I'm seeing "COPY FROM copies data from a file to a table" at http://www.postgresql.org/docs/current/static/sql-copy.html ... might work? I don't know. This would be to get the contents of her file into the setting table. If anyone has experience in this area, please advise.

Once upon a time the setting table itself was limited to 1024 characters but we changed this in 44e4661 as part of #972 to store the "General Terms of Use" we're talking about. Maybe we should store large values like this somewhere else, as you're suggesting, @bencomp .

[bencomp]

The COPY command is for importing/exporting tabular data from/to files, so that wouldn't work. Unless it is critical to put the terms in place right now, I would advise to wait for a fix.

Not writing settings changes to the action log goes defeats the purpose of the action log (which I initially called an "audit log").

From the service provider's point of view, keeping the terms in a version control system is essential, in case of an audit or a dispute.

[ajirnyi]

Hi all -- just wondering, is there any currently functional workaround for this issue, or is the only solution to place the ToC off-site with a link?

Thanks!!

[pdurbin]

@ajirnyi @shlake on my Mac, I just put the contents of the failing file (more that 1024 characters) in my copy/paste buffer from the terminal like this...

curl https://github.com/IQSS/dataverse/files/18657/Scratch2.html.txt | pbcopy

... (but there are other ways, of course) and used pgAdmin3 to create a new "name" in the "setting" tabled called ":ApplicationTermsOfUse" and pasted from my buffer into the "content" column like this:

It seemed to work fine. It shows the content on sign up:

Obviously this is not the most elegant workaround so if @scolapasta @kcondon @landreev or others have something better we can document for now, please do leave a comment.

[shlake]

The character limit problem also shows up in setting (new as of 4.3) "DatasetPublishPopupCustomText". The following fails:
curl -X PUT -d "By clicking Continue you affirm that you have met all licenses, policy and other data agreements:<ulu have the legal right and authorization to make the data publicly available online world-wide through Libra</li><li>In preparing the data for public archiving and distribution, you have removed any confidential or sensitive information, student education records protected under FERPA, and all information that personally identifies any individual or that contains any information classified as highly sensitive under state or federal law, or UVA policy.</li><li>If the submission is based upon work that has been sponsored or supported by an agency or organization other than UVA such as the National Institutes of Health, the National Science Foundation, or a private sponsor or funder, you represent that you have fulfilled any right of review, confidentiality, or other obligations required by that contract or agreement.</li><li>You represent that you have made a reasonable effort to ensure that the data contained in your submission is accurate.</li><li>You represent that you have appropriately acknowledged other researchers whose work contributed to the data.</li></ul>" http://localhost:8080/api/admin/settings/:DatasetPublishPopupCustomText

But a shortened text string works.

[pdurbin]

@shlake does the PgAdmin3 trick I mentioned at #2669 (comment) help?

[shlake]

The work around we used was a regular SQL update command for each setting we couldn't get to work through the API (DatasetPublishPopupCustomText and ApplicationTermsOfUse). I wasn't the one to do it, if you need more info, I'll get it.

Update setting
Set content = 'the text you want to put’
Where name = ':DatasetPublishPopupCustomText’;

[pdurbin]

limiting the size of the text that is put in the action log

This is what we should do. Truncate it.

[pdurbin]

That said, people don't seem to be complaining about this very much. Closing.

[pdurbin]

Related: #4835

[pdurbin]

Related: pull request #5269.

[pdurbin]

@shlake @ajirnyi heads up that pull request #5269 was merged so you might want to try again.