Zitadel Executing System API Call

[HungryHowies]

Having issues using the system API.

Environment

• Ubuntu-22.0.4
• Zitadel-v2.50.0 ( tried other versions, v2.48.0, v2.49.0)
• nginx/1.18.0

My goal is simple. To be able to execute the following command.

curl -L -X POST 'https://$CUSTOM-DOMAIN/system/v1/instances/254409540550793937/domains' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
--data-raw '{
  "domain": "new-domain"
}'

From this documentation.

https://zitadel.com/docs/apis/resources/system/system-service-add-domain#adds-a-domain-to-an-instance

What I have done

Documentation Used:

https://zitadel.com/docs/guides/integrate/zitadel-apis/access-zitadel-system-api#system-api-user

Generate an RSA private key with 2048 bit modulus:

openssl genrsa -out system-user-1.pem 2048

and export a public key from the newly created private key:

openssl rsa -in system-user-1.pem -outform PEM -pubout -out system-user-1.pub

Results:

Provide the public key to the ZITADEL runtime configuration.

The path to the key:

SystemAPIUsers:
  - system-user-1:
      Path: /system-user-1.pub

Configured default.yml run-time file.

Results:

SystemAPIUsers:
# # Add keys for authentication of the systemAPI here:
# # you can specify any name for the user, but they will have to match the `issuer` and `sub` claim in the JWT:
 - system-user-1:
     Path: /usr/local/bin/system-user-1.pub 

Restart Zitadel

systemctl restart zitadel

Execute sysytem API and receive the following error.

root@zitadel-build:/usr/local/bin# curl -L -X POST 'https://zitadel-build.domain.com/system/v1/instances/254409540550793937/domains' -H 'Content-Type: application/json' -H 'Accept: application/json' --data-raw '{
  "domain": "new-domain"
}'
{"code":16, "message":"auth header missing"}root@zitadel-build:/usr/local/bin# 

Think I need to add bearer token . So, I created a machine user and gave roles.

From the documentation:

If you don't specify any memberships, you are allowed to access the whole ZITADEL System API.

So I'm not sure what s going on at this point.

My next try with system API call as follow:

 curl -L -X POST 'https://zitadel-build.domain.com/system/v1/instances/254409540550793937/domains' \
-H 'Accept: application/json' \
-H 'Authorization: Bearer -iUA_COZd-VjDEDeIKvnnYmASe3VIlUD-UAFwu5BdzZJBVNgs7odA27QrfGxOUxwTHSQqbE' \
--data-raw '{"domain": "new-domain"}' | jq

Results:

root@zitadel-build:/usr/local/bin# curl -L -X POST 'https://zitadel-build.domain.com/system/v1/instances/254409540550793937/domains' -H 'Accept: application/json' -H 'Authorization: Bearer -iUA_COZd-VjDEDeIKvnnYmASe3VIlUD-UAFwu5BdzZJBVNgs7odA27QrfGxOUxwTHSQqbE' --data-raw '{"domain": "new-domain"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100   196  100    24   1105    135 --:--:-- --:--:-- --:--:--  1242
{
  "code": 7,
  "message": "No matching permissions found (AUTH-5mWD2)",
  "details": [
    {
      "@type": "type.googleapis.com/zitadel.v1.ErrorDetail",
      "id": "AUTH-5mWD2",
      "message": "No matching permissions found"
    }
  ]
}
root@zitadel-build:/usr/local/bin# 

ZITADEL Tools

Further down the documentation I tried Zitadel-tools

root@zitadel-build:/usr/local/bin# zitadel-tools key2jwt --audience=https://zitadel-build.domain.com --key=system-user-1.pem --issuer=system-user-1
2024/04/16 17:52:08 error generating jwt: x509: failed to parse private key (use ParsePKCS8PrivateKey instead for this key format)

So far at this point none of mine system API calls work, BUT on the upside this does work but its not a system API call.

root@zitadel-build:/home/ubuntu# curl -L -X POST 'https://zitadel-build.domain.com/admin/v1/domains/_search' -H 'Accept: application/json' -H 'Authorization: Bearer wf0zBpc4kmPKk6j_d-ASEF6FAj3Mr2dyw-ubHwgsn3W3G7aLoMLbW3TIbbsho7n3dsgdRug' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   921  100   921    0     0   2764      0 --:--:-- --:--:-- --:--:--  2774
{
  "details": {
    "totalResult": "4",
    "viewTimestamp": "2024-04-10T22:11:03.510701Z"
  },
  "result": [
    {
      "details": {
        "sequence": "156",
        "creationDate": "2024-04-02T23:12:09.742141Z",
        "changeDate": "2024-04-02T23:12:09.742141Z",
        "resourceOwner": "254409540550793937"
      },
      "domain": "localhost"
    },
    {
      "details": {
        "sequence": "155",
        "creationDate": "2024-04-02T23:12:09.742141Z",
        "changeDate": "2024-04-02T23:12:09.742141Z",
        "resourceOwner": "254409540550793937"
      },
      "domain": "startrek-04yj0z.localhost",
      "generated": true
    },
    {
      "details": {
        "sequence": "104",
        "creationDate": "2024-02-17T02:11:44.636411Z",
        "changeDate": "2024-02-17T02:11:44.636411Z",
        "resourceOwner": "254409540550793937"
      },
      "domain": "startrek-04yj0z.zitadel-build.domain.com",

Im trying to understand where i may have went wrong. Any help would be apperciated.

[HungryHowies]

Another configuration test. No joy.

I adjust my SystemAPIUser settings as follow.

SystemAPIUsers:
# # Add keys for authentication of the systemAPI here:
# # you can specify any name for the user, but they will have to match the `issuer` and `sub` claim in the JWT:
 - system-user-1:
      Path: "/usr/local/bin/system-user-1.pub"  # you can provide the key either by reference with the path
      Memberships:
#       # MemberType System allows the user to access all APIs for all instances or organizations
        - MemberType: System
          Roles:
            - "SYSTEM_OWNER"
#           # Actually, we don't recommend adding IAM_OWNER and ORG_OWNER to the System membership, as this basically enables god mode for the system user
            - "IAM_OWNER"
            - "ORG_OWNER"
#       # MemberType IAM an

Stopped Zitadel service,

systemctl stop zitadel

Then i execute setup

zitadel setup  --init-projections=true   --config defaults.yaml --steps steps.yaml --masterkey "MasterkeyNeedsToHave32Characters"  --tlsMode external

No errors were shown.

Started Zitadel service.

systemctl  start zitadel

Still have error

{
  "code": 7,
  "message": "No matching permissions found (AUTH-5mWD2)",
  "details": [
    {
      "@type": "type.googleapis.com/zitadel.v1.ErrorDetail",
      "id": "AUTH-5mWD2",
      "message": "No matching permissions found"
    }
  ]
}

Im running out of ideas what it could be. All the other API calls work except for system API's

[fforootd]

Hm weird that error.

You called the system api with a token created by zitadel-tools?

[HungryHowies]

You called the system api with a token created by zitadel-tools?

I was unable to get Zitadel-tools to work.

Unsure about this part here ,

For troubleshooting, I created my machine users with the same name as my SYSTEM_USER ( i.e., system-user-1) then I downloaded the key as shown here.

What I did notice in the "Generate JWT" step, my key I downloaded showed that information. So I'm confused at this point, was I suppose to do that ? The documentation did not state to download a key from Web UI.

Here is my key I downloaded from user "system-user-1". I created this in the Web UI.

root@zitadel-build:/usr/local/bin# cat test.json

{"type":"serviceaccount","keyId":"263117051483134673","key":"-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAsyYwCkaMyCNs64iTMVwymU4hn3LQ04YyXoqYyYaxG128B0PH\nF73EOkuxICq0OzV9EK546FtQQoBfnXYM3D0ol+C5KUSopdRZ3E5bywOFtPCqSj+p\nK9epuKU6nkNVpiBkp+O/jZh7+w0divEjJiwQqSy1cZd6dBGZ6MUM4esrANJsiq3I\niAfrWOcSinpS9KndS/69kWTKyV154FrKbfiXNYrYgAuFoyU34AxVNTuCbY4ZQ41u\nUKXoGZzxRgqnsIIJZHi81tN5Z1kk71RVzaT8K9WZfLbfZbrWsxXwNWsiTcj4vX9T\nfuQ1xZjIYaw3LsAU+3ON2aLEB2YgtlPv89D5wT9VP53pK8fQosgGGOpYp1KpCSF16m5dct\nD7jWh15nlGFSSO9VpKc4d+UCgYEAtxuM2r3g/jRFPowvAiXIpaVOMypR9Lwnf9Od\nbLM/BQjGhXwu..................................................QBWytoEJjKFdUHnihVKPQIn6R16YMeUVRnFeGTE+02MgU8eYEx0Q\n2EziXUfugwInrF46uFwkNeZfd9mhIBHqRCYU5p68mZSeia33m+4R6e6B6kkpbkT5\nE9TwnbkCgYBa3saqlmiIF2vPcjw7rAGfh+NQplrqNUxS9VW0HfsyoaJ/E+BvuLJ8\nMTKfFrlyUXztPP3w+x9z1bwPl5HoAeGhgqUp+5OVg/8DSuKarAGSji2DExX4Yl5N\nFKU0/0dvIbTNeuQcEOwyDOBRAFWvyib11Lj9IHVR09gDLrCay3+Hfw==\n-----END RSA PRIVATE KEY-----\n","expirationDate":"2024-06-12T05:00:00Z","userId":"263116887049640657"}

Just to clarify, another Machine user I created works on every API call except for /SYSTEM which I understand now. The step in which I need to get/make the token is where I'm failing at. This is just a guess.

[HungryHowies]

@fforootd

Ok now I see where I'm failing.

you can specify any name for the user, but they will have to match the issuerandsub claim in the JWT

The JWT payload will need to contain the following claims:
Some where this need to happen.

{
  "iss": "system-user-1",
  "sub": "system-user-1",
  "aud": "https://custom-domain.com",
  "iat": 1659957184,
  "exp": 1659960784
}

I need to get the Zitadel-tools to work. This it where it fails, I'm not sure if I need to make Action or modify my RSA keypair from the doc's. I have tried both still getting the same error as shown above.

EDIT:
No matter what I do it seams that Zitadel-Tool doesn't work for me.
Following the documentation.

Execute this command after creating certs.

zitadel-tools key2jwt --audience=https://zitadel-build.domain.comm --key=system-user-1.pem  --issuer=system-user-1 

Output:

2024/04/17 20:20:34 error generating jwt: x509: failed to parse private key (use ParsePKCS8PrivateKey instead for this key format)

I think its the way the certs are made openssl genrsa -out system-user-1.pem 2048 , Not 100% sure.

[HungryHowies]

@fforootd

Found something, first this post

#6576 (comment)

From what livio-a stated I tried openssl genrsa --traditional 2048

The results doesn't look right.

I executed the the following command

root@zitadel-build:/usr/local/bin# zitadel-tools key2jwt --audience=https://zitadel-build.domain.com  --key=system-user-1.pem --issuer=system-user-1

Output:

eyJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJzeXN0ZW0tdXNlci0xIiwic3ViIjoic3lzdGVtLXVzZXItMSIsImF1ZCI6WyJodHRwczovL3ppdGFkZWwtYnVpbGQuaHVuZ3J5LWhvd2FyZC5jb20iXSwiaWF0IjoxNzEzNDExNDU2LCJleHAiOjE3MTM0MTUwNTZ9.X8wupidnGUwj0TWQD65a8dMsgMJnDvTLaNr-9sR3B33IzI9QDk6F-GzpTbWjF7HD0dMjN-O3tjTQVBxc21BmIOpGSHAtNhBgKlTf7qTe9z28saXxgNaElSyZCtVaUpgPofnWSmhr7-aejOTwKto4aOfREg9KQerRuF-HfGXuljf_cuARuiINZ6eU9vkz_2IKWffhlNWyME9xbG6SfiWym9HheInRDUZFoWX56NgDadf_7bg140zMMhOGfZfcupP7d1NqO-_stMCMVCH0zivjKGvpIR04Kh-V00HLRDkZdv4MScRpxuA0GVO594n3ez0XgDT_sT7qeGKRSc_qvXuAyw

So I tried again with a SYSTEM API

curl -L -X POST 'https://zitadel-build.domain.com/system/v1/instances/254409540550793937/domains' -H 'Content-Type: application/json' -H 'Accept: application/json' --data-raw '{
  "domain": "new-domain"
}' |jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    72  100    44  100    28    517    329 --:--:-- --:--:-- --:--:--   857
{
  "code": 16,
  "message": "auth header missing"
}

Different error, I cant win... 😆

How I found that post was from here.

zitadel/zitadel-tools#97

[fforootd]

Hm in the call above you miss the authorization header (i.e. the token you generated needs to be sent as such)

[HungryHowies]

Hm in the call above you miss the authorization header (i.e. the token you generated needs to be sent as such)

Understood, but I'm not sure how to make the token from Zitadel-tools, its just not working as shown above.

[fforootd]

hm zitadel-tools will just create the token for you, you can then send that token as authorization: bearer {token} in the curl example

[HungryHowies]

Agree, but Zitadel-tools is failing to work as shown below. Not sure how to go about it

root@zitadel-build:/usr/local/bin# zitadel-tools key2jwt --audience=https://zitadel-build.domain.com --key=system-user-1.pem --issuer=system-user-1
2024/04/16 17:52:08 error generating jwt: x509: failed to parse private key (use ParsePKCS8PrivateKey instead for this key format)

[HungryHowies]

@fforootd

I found my issue why Zitadel-tools failed. zitadel/zitadel-tools#97

[fforootd]

Oh wow... I was not aware of that, sorry sir!

[HungryHowies]

No worries, I don't know much about how to use GO, but I wanted to learn how Zitadel-tools actually works in making private key into JWT , I did notice something about cobra in there. It may take a few to understand the "HowTo" in doing this.