Problems with REST API tokens in cluster

[hijbas]

When using the REST API in a clustered Graylog setup, it seems like tokens only works when communicating with the Graylog master node. Communication with other nodes in the cluster is only possible using a username/password or a session token.

I've tested listing the nodes with /api/cluster (see below) but also information about a specific node with /api/cluster/{node_id}/jvm gives similar errors. Requesting information about master node is OK, but I get ApiError/Unauthorized when requesting information about other Graylog nodes.

Expected Behavior

Expected to see information about all nodes. This also works as expected when using a username/password or session token, but not a "permanent token" generated using the web UI.

Current Behavior

Only information about the master server is shown, like below.

curl -u 1h0kc6im5h16r5qbin9vs9l0d9vud35k06ld2ogjehgeg4mq3qi0:token -X GET http://172.17.11.9:9000/api/cluster?pretty=true
{
  "c317593f-[...]-b0b5b57da06b" : {
    "facility" : "graylog-server",
    "codename" : "Noir",
    "node_id" : "c317593f-[...]-b0b5b57da06b",
    "cluster_id" : "cac50c4a-[...]-8f8d64d831dd",
    "version" : "4.1.10+9bc6267",
    "started_at" : "2022-02-01T15:30:28.670Z",
    "hostname" : "ulm-graylog-master",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Etc/UTC",
    "operating_system" : "Linux 3.10.0-1160.49.1.el7.x86_64",
    "is_processing" : true
  },
  "ff94b945-[...]-2aca679115f5" : null,
  "101c03c9-[...]-9026c41bb5d2" : null
}

Your Environment

• Graylog Version: 4.1.10 but also have same problem in 3.3.16. A co worker says this has worked in 3.2, but I haven't been able to verify this.
• Java Version: 1.8.0_312
• Elasticsearch Version: 6.8.21
• MongoDB Version: 4.2
• Operating System: RedHat Linux 7.9
• Browser version: curl 7.29.0

Running Graylog docker images (graylog/graylog:4.1.10) in Kubernetes. Mongo and Elasticsearch is also running in Kubernetes. Graylog is clustered with one master node and two worker nodes.

[manderio]

I am having a similar issue, though my token doesn't prompt a response at all. When examining the logs I get:

Realm [org.graylog2.security.realm.SessionAuthenticator@3e3a44ac] does not support token
AccessTokenAuthToken{hashcode=[hashcode], host=[hostname]}.  Skipping realm

I redacted some info regarding out setup because policy.

I have looked through about anything, but I have no idea what is causing this. Our setup is mildly different though, we have a cluster set up on a host, not in a container setup, we are using graylog version 4.5 and this is a new install on ubuntu. It seems as if the API got issues with token authentication specifically. When using a username and password in curl, the authentication goes through fine.

I don't know if this is a separate issue, but as it seems related, I decided to jump onto this thread. Please let me know if I should report this in a separate issue.

[thll]

I'm unable to reproduce the problem outlined in the issue description.

If the issue still exists, could you please check the Graylog server log on all nodes when you run the request against /api/cluster on the leader node?
Also, what happens if you request /api/cluster on a follower-node, not the leader node?

[thll]

I'm closing this for now because we were unable to reproduce this. Please re-open if the issue reoccurs.