Open OnDemand Internal Server Error #184
Description
When trying to open the OnDemand URL: https://localhost:3443 in a browser I get the following error:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
The error appears in the ondemand logs (docker compose logs -f ondemand) as:
ondemand | Completed successfully!
ondemand | ---> Starting ondemand-dex...
ondemand | ---> Starting ondemand httpd24...
ondemand | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.19.0.12. Set the 'ServerName' directive globally to suppress this message
ondemand | time="2024-07-25T14:22:47Z" level=info msg="Dex Version: v2.36.0, Go Version: go1.19.2, Go OS/ARCH: linux amd64"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="config issuer: https://localhost:5554"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="config storage: sqlite3"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="config static client: OnDemand"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="config connector: ldap"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="config skipping approval screen"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="config refresh tokens rotation enabled: true"
ondemand | time="2024-07-25T14:22:47Z" level=info msg="keys expired, rotating"
ondemand | time="2024-07-25T14:22:48Z" level=info msg="keys rotated, next rotation: 2024-07-25 20:22:47.998753652 +0000 UTC"
ondemand | time="2024-07-25T14:22:48Z" level=info msg="listening (telemetry) on 0.0.0.0:5558"
ondemand | time="2024-07-25T14:22:48Z" level=info msg="listening (http) on 0.0.0.0:5556"
ondemand | time="2024-07-25T14:22:48Z" level=info msg="listening (https) on 0.0.0.0:5554"
ondemand | 2024/07/25 14:23:58 http: TLS handshake error from [::1]:52182: local error: tls: bad record MAC
ondemand | 2024/07/25 14:39:17 http: TLS handshake error from [::1]:51144: local error: tls: bad record MAC
ondemand | 2024/07/25 14:42:10 http: TLS handshake error from [::1]:40056: local error: tls: bad record MAC
I'm running the containers in Pop OS 20.04. XDMoD and Coldfront interfaces work correctly. The Open OnDemand interface had previously worked as recent as 2 weeks ago. I have tried running ./hpcts destroy and ./hpcts cleanup as well as cloning a new repository and running the setup again. My colleague can also reproduce the error.
The fact that it had previously worked leads me to believe it could be an expired certificate maybe?