migration 0.11.10 > 1.0 : did you forget to run hydra migrate sql" or forget to set the SYSTEM_SECRET

[JamesnaW]

I've try to migrate to new version, while I run hydra hydra serve --dangerous-force-http it's always fails with
INFO[0000] Connected to SQL! FATA[0000] Could not fetch private signing key for OpenID Connect - did you forget to run "hydra migrate sql" or forget to set the SYSTEM_SECRET? error="unexpected end of JSON input"

But if I create new db schema it's working fine.

[aeneasr]

To which version of 1.0 are you upgrading? Did you set the SYSTEM_SECRET?

[JamesnaW]

1.0.0-beta.6 and I already set SYSTEM_SECRET. Its the same as I using in version 0.11.10.
SYSTEM_SECRET="ASLDHIUqwhdfsjc2345ute0rgQ!R@WTQ(shgj"

[aeneasr]

Could you try upgrading to beta.4 and see if that works?

[aeneasr]

So the thing is, "unexpected end of JSON input" points to a malformed JSON string in the table. I'm not sure how that can happen here. The store works as follows:

1. Marshall the key as JSON string
2. Encrypt the resulting string with SYSTEM_SECRET and AES-GCM
3. Store the resulting byte array in the database

To retrieve it, it works as follows:

1. Retrieve the encrypted byte array from the database
2. Decrypt the byte array using SYSTEM_SECRET and AES-GCM (aka with integrity check)
3. If integrity check passes decode/unmarshall the byte array with json.Unmarshal
4. Return the resulting object

Since you're seeing unexpected end of json input, something weird is going on. First, the AES-GCM check passes. This means, that the encrypted string has been properly decrypted, otherwise the integrity check would fail. This also means that the decrypted string is somewhat JSON parsable as we're seeing an error that points to the end of the string, not the beginning - implying that the string starts, at least, with something like {.

Not sure what's going on here. One way to solve this without negative side effects is to remove the key from the database manually:

DELETE FROM hydra_jwk WHERE sid='hydra.openid.id-token'

[JamesnaW]

Thanks for quick reply. I have try to use beta.4 and it fail too. So I try DELETE FROM hydra_jwk WHERE sid='hydra.openid.id-token' and it work. By the way is it will has affect to the system?

[aeneasr]

No effect whatsoever, unless you hardcoded that JSON Web Key somewhere (which you shouldn't and probably didn't). All the command does is to remove the old key. When restarting hydra, the key will be automatically re-generated. There's also a bug in #921 which causes the beta.6 upgrade to fail if a pre-existing key is stored in the database but that is - as far as I can tell - unrelated to what you're experiencing.

[JamesnaW]

Thanks @arekkas .