portal list fails with 500 service failure

[mbjones]

Describe the bug
Accessing https://search.dataone.org/portals produces two service errors, and does not show the list of portals.

In discussion today, we speculated that it might be related to the number of groups I am a part of, or something like that, as it was not happening for other people. From the network inspector, here is one of the errors that is producing a 500 ERROR response:

curl 'https://mn-ucsb-1.dataone.org/knb/d1/mn/v2/query/solr/?q=(-(rightsHolder:(%22CN%3Ddataone%5C-coredev%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DNCEAS%5C-interns%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DSASAP%20Data%20Team%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DDBO%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Dknb%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DDataONE%5C-Support%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DLikely%20Suspects%20Framework%20Users%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DSASAP%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Darctic%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Dcerp%5C-sfwmd%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22http%5C%3A%5C%2F%5C%2Forcid.org%5C%2F0000%5C-0003%5C-0077%5C-4738%22%20OR%20%22CN%3DMatt%20Jones%20A729%2CO%3DGoogle%2CC%3DUS%2CDC%3Dcilogon%2CDC%3Dorg%22%20OR%20UID%3Djones%2CO%3DNCEAS%2CDC%3Decoinformatics%2CDC%3Dorg)%20OR%20writePermission:(%22CN%3Ddataone%5C-coredev%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DNCEAS%5C-interns%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DSASAP%20Data%20Team%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DDBO%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Dknb%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DDataONE%5C-Support%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DLikely%20Suspects%20Framework%20Users%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DSASAP%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Darctic%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Dcerp%5C-sfwmd%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22http%5C%3A%5C%2F%5C%2Forcid.org%5C%2F0000%5C-0003%5C-0077%5C-4738%22%20OR%20%22CN%3DMatt%20Jones%20A729%2CO%3DGoogle%2CC%3DUS%2CDC%3Dcilogon%2CDC%3Dorg%22%20OR%20UID%3Djones%2CO%3DNCEAS%2CDC%3Decoinformatics%2CDC%3Dorg)%20OR%20changePermission:(%22CN%3Ddataone%5C-coredev%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DNCEAS%5C-interns%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DSASAP%20Data%20Team%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DDBO%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Dknb%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DDataONE%5C-Support%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DLikely%20Suspects%20Framework%20Users%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3DSASAP%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Darctic%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22CN%3Dcerp%5C-sfwmd%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22%20OR%20%22http%5C%3A%5C%2F%5C%2Forcid.org%5C%2F0000%5C-0003%5C-0077%5C-4738%22%20OR%20%22CN%3DMatt%20Jones%20A729%2CO%3DGoogle%2CC%3DUS%2CDC%3Dcilogon%2CDC%3Dorg%22%20OR%20UID%3Djones%2CO%3DNCEAS%2CDC%3Decoinformatics%2CDC%3Dorg))%20AND%20formatId:*dataone.org%5C%2Fportals*%20AND%20-obsoletedBy:*)&fl=id,seriesId,title,formatId,label,logo,datasource,writePermission,changePermission,rightsHolder,abstract&sort=dateUploaded%20desc&rows=99999&start=0&wt=json' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Authorization: Bearer $TOKEN' -H 'Origin: https://search.dataone.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: https://search.dataone.org/' -H 'Cookie: JSESSIONID=F8F2760555EA8AAAF6F6C9C82B6BB8E8; _ga=GA1.2.1631159309.1637876205' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' -H 'Sec-GPC: 1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache'

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://search.dataone.org/portals'
2. Note the list should show if not logged in
3. Log in {maybe as a user with a lot of groups}
4. See errors in the screen and network panel indicating 500 HTTP responses

Expected behavior
Portal list should display wothout errors.

Screenshots

Desktop (please complete the following information):

• OS: MacOS 11.6.6
• Browser FF
• Version 103.0.1

[robyngit]

@mbjones I am not able to reproduce this. I tried forcing the portal search filters to use all of your group IDs (by hardcoding in the values in Filters.addWritePermissionFilter), but was able to get a list of results without error. Similarly, the curl command you shared is working for me. Are there any other details that you can share that might help enable me to reproduce the error?

For future reference, here are the subject IDs I used:

[
  "CN=dataone\-coredev,DC=dataone,DC=org",
  "CN=NCEAS\-interns,DC=dataone,DC=org",
  "CN=SASAP Data Team,DC=dataone,DC=org",
  "CN=DBO,DC=dataone,DC=org",
  "CN=knb\-data\-admins,DC=dataone,DC=org",
  "CN=DataONE\-Support,DC=dataone,DC=org",
  "CN=Likely Suspects Framework Users,DC=dataone,DC=org",
  "CN=SASAP,DC=dataone,DC=org",
  "CN=arctic\-data\-admins,DC=dataone,DC=org",
  "CN=cerp\-sfwmd\-admins,DC=dataone,DC=org",
  "http\:\/\/orcid.org\/0000\-0003\-0077\-4738",
  "CN=Matt Jones A729,O=Google,C=US,DC=cilogon,DC=org",
  "UID=jones,O=NCEAS,DC=ecoinformatics,DC=org"
]

[mbjones]

More details. From the developer Network tab, I see a 500 error on two SOLR queries during the page load. The first is for this request URI:

SOLR URI request

- https://mn-ucsb-1.dataone.org/knb/d1/mn/v2/query/solr/?q=%28%28rightsHolder%3A%28%22CN%3Ddataone%5C-coredev%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DNCEAS%5C-interns%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DSASAP+Data+Team%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DDBO%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Dknb%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DDataONE%5C-Support%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DLikely+Suspects+Framework+Users%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DSASAP%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Dcerp%5C-sfwmd%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Darctic%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22http%5C%3A%5C%2F%5C%2Forcid.org%5C%2F0000%5C-0003%5C-0077%5C-4738%22+OR+%22CN%3DMatt+Jones+A729%2CO%3DGoogle%2CC%3DUS%2CDC%3Dcilogon%2CDC%3Dorg%22+OR+UID%3Djones%2CO%3DNCEAS%2CDC%3Decoinformatics%2CDC%3Dorg%29+OR+writePermission%3A%28%22CN%3Ddataone%5C-coredev%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DNCEAS%5C-interns%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DSASAP+Data+Team%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DDBO%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Dknb%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DDataONE%5C-Support%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DLikely+Suspects+Framework+Users%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DSASAP%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Dcerp%5C-sfwmd%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Darctic%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22http%5C%3A%5C%2F%5C%2Forcid.org%5C%2F0000%5C-0003%5C-0077%5C-4738%22+OR+%22CN%3DMatt+Jones+A729%2CO%3DGoogle%2CC%3DUS%2CDC%3Dcilogon%2CDC%3Dorg%22+OR+UID%3Djones%2CO%3DNCEAS%2CDC%3Decoinformatics%2CDC%3Dorg%29+OR+changePermission%3A%28%22CN%3Ddataone%5C-coredev%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DNCEAS%5C-interns%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DSASAP+Data+Team%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DDBO%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Dknb%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DDataONE%5C-Support%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DLikely+Suspects+Framework+Users%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3DSASAP%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Dcerp%5C-sfwmd%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22CN%3Darctic%5C-data%5C-admins%2CDC%3Ddataone%2CDC%3Dorg%22+OR+%22http%5C%3A%5C%2F%5C%2Forcid.org%5C%2F0000%5C-0003%5C-0077%5C-4738%22+OR+%22CN%3DMatt+Jones+A729%2CO%3DGoogle%2CC%3DUS%2CDC%3Dcilogon%2CDC%3Dorg%22+OR+UID%3Djones%2CO%3DNCEAS%2CDC%3Decoinformatics%2CDC%3Dorg%29%29+AND+formatId%3A*dataone.org%5C%2Fportals*+AND+-obsoletedBy%3A*%29&fl=id%2CseriesId%2Ctitle%2CformatId%2Clabel%2Clogo%2Cdatasource%2CwritePermission%2CchangePermission%2CrightsHolder%2Cabstract&sort=dateUploaded+desc&rows=99999&start=0&wt=json

The error response provides a clue: Request Header Fields Too Large:

<error detailCode="Solr server error" errorCode="500" name="ServiceFailure">
    <description>Error from server at http://localhost:8983/solr/metacat-index: Expected mime type application/octet-stream but got text/html. &lt;h1&gt;Bad Message 431&lt;/h1&gt;&lt;pre&gt;reason: Request Header Fields Too Large&lt;/pre&gt;</description>
</error>

When I look at my request headers, the only large header is the Authorization header with my JWT token in it. Maybe the request URI itself is too long with all of the groups? I do note that the solr request by URI itself resolves fine outside of the page via a curl command.

[robyngit]

I believe this issue is an instance of #1712; related to #1998

Here is how I am able to reproduce this issue:

1. Create a profile with many groups with long names (~ more than 12)
2. In the app config, set the disableQueryPOSTs option to true

disableQueryPOSTs defaults to false, but for the DataONE theme it's set to true.

@mbjones can you confirm that you do not see this issue on the KNB (where disableQueryPOSTs is false)? Do you know why we have disabled solr POSTs for the DataONE MetacatUI?

[mbjones]

Good catch, @robyngit -- that makes perfect sense. And you're right, it works fine for me on the KNB site.

The disableQueryPOSTs=false was was added to fix problems with long URLs. Maybe the ticket in which this was implemented would have details as to why its not configured on all sites. Seems to me like we should be using it everywhere to handle these issues, unless it introduces some sort of problem.

[robyngit]

In reference to the (new) disableQueryPOSTs config option in March 2019, Lauren said:

The DataONE theme will likely have this off for a while until POST is supported on the CN.

It appears that POST is now supported on the CN, so I updated the DataONE theme to allow queries using POST.