Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER #7071

k-e-r-n-e-l-p-a-n-i-c · 2022-09-16T12:42:39Z

Environment info

NooBaa Version: 5.10.1
Platform: OpenShift 4.8

Actual behavior

noobaa api events_api read_activity_log '{}' fails with error as provided in More Information

Expected behavior

Activity logs are received as a response.

Steps to reproduce

noobaa api events_api read_activity_log '{}'

More information - Screenshots / Logs / Other output

INFO[0008] ✅ Exists: NooBaa "noobaa"
INFO[0008] ✅ Exists: Service "noobaa-mgmt"
INFO[0008] ✅ Exists: Secret "noobaa-operator"
INFO[0009] ✅ Exists: Secret "noobaa-admin"
INFO[0014] ✈️ RPC: events.read_activity_log() Request: map[]
WARN[0014] RPC: GetConnection creating connection to wss://localhost:45005/rpc/ 0xc000478d80
INFO[0014] RPC: Connecting websocket (0xc000478d80) &{RPC:0xc0003d9c20 Address:wss://localhost:45005/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:}
INFO[0016] RPC: Connected websocket (0xc000478d80) &{RPC:0xc0003d9c20 Address:wss://localhost:45005/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:}
ERRO[0018] ⚠️ RPC: events.read_activity_log() Response Error: Code=INVALID_SCHEMA_REPLY Message=INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log
FATA[0018] ❌ INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log

We tried 'read_bucket' of 'buckets_api' and that works without any issues when executed from CLI.

As a part of our automation we wanted to get activity logs of obj.upload and obj.download events. Please let me know if this is the correct approach or I am missing something here?

k-e-r-n-e-l-p-a-n-i-c · 2022-09-24T09:32:47Z

Hi,

Can you please provide some help on this issue?

baum · 2022-10-02T12:27:06Z

Hello @arunprasadmudaliar 🖖

I have tried to reproduce this issue with both v5.10.1 and the latest master branch operator/core installation, I do not see the issue:

➜  noobaa-operator git:(master) ✗ ./build/_output/bin/noobaa-operator-local -n noobaa api events_api read_activity_log '{}'
INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
INFO[0000] ✅ Exists: Secret "noobaa-admin"
INFO[0000] ✈️  RPC: events.read_activity_log() Request: map[]
WARN[0000] RPC: GetConnection creating connection to wss://localhost:65366/rpc/ 0xc0003b6b40
INFO[0000] RPC: Connecting websocket (0xc0003b6b40) &{RPC:0xc00045dd60 Address:wss://localhost:65366/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0000] RPC: Connected websocket (0xc0003b6b40) &{RPC:0xc00045dd60 Address:wss://localhost:65366/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}
INFO[0000] ✅ RPC: events.read_activity_log() Response OK: took 44.9ms
logs:
- desc:
  - Drive /noobaa_storage of noobaa-default-backing-store-noobaa-pod-e25426c8 in pool
    noobaa-default-backing-store is online
  event: node.connected
  id: 63397dee5c53f6002fc965a0
  level: info
  node:
    linkable: true
    name: noobaa-default-backing-store-noobaa-pod-e25426c8#1
    pool: noobaa-default-backing-store
  time: 1664712174965
- actor:
    email: operator@noobaa.io
    linkable: true
  desc:
  - A managed pool noobaa-default-backing-store with 1 nodes was created by operator@noobaa.io
  event: resource.create
  id: 63397d225c53f6002fc9659c
  level: info
  pool:
    linkable: true
    name: noobaa-default-backing-store
    resource_type: HOSTS
  time: 1664711970827
- account:
    email: admin@noobaa.io
    linkable: true
  desc:
  - 'admin@noobaa.io was created '
  event: account.create
  id: 63397cc45c53f6002fc96593
  level: info
  time: 1664711876291
- actor:
    email: admin@noobaa.io
    linkable: true
  desc:
  - noobaa was created by admin@noobaa.io
  event: conf.create_system
  id: 63397cb65c53f6002fc9658f
  level: info
  time: 1664711861995

Could you please provide more info?
Also, it might be a good idea to verify that the CLI and the core versions match.

v5.10.1

➜  noobaa-operator git:(6f571fd) ✗  ./build/_output/bin/noobaa-operator-local version
INFO[0000] CLI version: 5.10.1
INFO[0000] noobaa-image: noobaa/noobaa-core:5.10.1
INFO[0000] operator-image: noobaa/noobaa-operator:5.10.1
➜  noobaa-operator git:(6f571fd) ✗ k get  noobaa noobaa -n noobaa -o yaml | grep Core
  readme: "\n\n\tWelcome to NooBaa!\n\t-----------------\n\tNooBaa Core Version:     5.10.1-e8c9b09\n\tNooBaa

Latest master

➜  noobaa-operator git:(master) ✗ ./build/_output/bin/noobaa-operator-local version
INFO[0000] CLI version: 5.12.0
INFO[0000] noobaa-image: noobaa/noobaa-core:master-20220913
INFO[0000] operator-image: noobaa/noobaa-operator:5.12.0
➜  noobaa-operator git:(master) ✗ k get  noobaa noobaa -n noobaa -o yaml | grep Core
  readme: "\n\n\tWelcome to NooBaa!\n\t-----------------\n\tNooBaa Core Version:     5.12.0-cfbf726\n\tNooBaa

Please share your thoughts.
Best regards.

k-e-r-n-e-l-p-a-n-i-c · 2022-10-04T04:22:34Z

Hello @baum,

Thank you for responding. Based on your comment, I verified the version of the client and the operator and indeed they were different in my case. ( operator was 5.8.0 and the client was 5.10.0).

I have now downgraded my client version to 5.8.0 to match the operator:

CLI
INFO[0000] CLI version: 5.8.0
INFO[0000] noobaa-image: noobaa/noobaa-core:5.8.0
INFO[0000] operator-image: noobaa/noobaa-operator:5.8.0

Operator version ( as shown in web-console)

But I still get the same error:

INFO[0002] ✅ Exists: NooBaa "noobaa"
INFO[0002] ✅ Exists: Service "noobaa-mgmt"
INFO[0002] ✅ Exists: Secret "noobaa-operator"
INFO[0003] ✅ Exists: Secret "noobaa-admin"
INFO[0003] ✈️ RPC: events.read_activity_log() Request: map[]
WARN[0003] RPC: GetConnection creating connection to wss://localhost:39703/rpc/ 0xc000f0f4a0
INFO[0003] RPC: Connecting websocket (0xc000f0f4a0) &{RPC:0xc00003a5a0 Address:wss://localhost:39703/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
INFO[0004] RPC: Connected websocket (0xc000f0f4a0) &{RPC:0xc00003a5a0 Address:wss://localhost:39703/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
ERRO[0024] ⚠️ RPC: events.read_activity_log() Response Error: Code=INVALID_SCHEMA_REPLY Message=INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log
FATA[0024] ❌ INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log

I have to connect to the server through a VPN connection and initially I thought might be its due to VPN. But later I realized that the websocket connection shows connected and as well as other api queries related to bucket works fine. It's only the events api that throws an error.

I tried the export_activity_log but again I get an error. Using the -o flag does not help here. Looks like it is failing due to type casting.

noobaa api events_api export_activity_log '{}' -o yaml
INFO[0005] ✅ Exists: NooBaa "noobaa"
INFO[0006] ✅ Exists: Service "noobaa-mgmt"
INFO[0006] ✅ Exists: Secret "noobaa-operator"
INFO[0006] ✅ Exists: Secret "noobaa-admin"
INFO[0008] ✈️ RPC: events.export_activity_log() Request: map[]
WARN[0008] RPC: GetConnection creating connection to wss://localhost:38625/rpc/ 0xc000521a90
INFO[0008] RPC: Connecting websocket (0xc000521a90) &{RPC:0xc000460690 Address:wss://localhost:38625/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
INFO[0011] RPC: Connected websocket (0xc000521a90) &{RPC:0xc000460690 Address:wss://localhost:38625/rpc/ State:init WS: PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}
ERRO[0069] ⚠️ RPC: events.export_activity_log() Call failed: json: cannot unmarshal string into Go struct field .reply of type map[string]interface {}
FATA[0069] ❌ json: cannot unmarshal string into Go struct field .reply of type map[string]interface {}

Please provide any suggestions that you want me to try. Upgrading the operator to latest version is bit difficult as it is maintained by another team.

baum · 2022-10-06T07:28:29Z

Hello @arunprasadmudaliar 🖖

I do see an issue with export_activity_log, see the 🧹 Fix event api export_activity_log reply schema #7087 PR.
Re read_activity_log, could you try with v5.10.1 CLI?

Best regards.

k-e-r-n-e-l-p-a-n-i-c · 2022-10-06T08:45:35Z

Hello @baum,

Thanks for sharing the above issue details for export_activity_log.

I tried to run read_activity_log with 5.10.1 cli but the error still remains the same.

ERRO[0009] ⚠️ RPC: events.read_activity_log() Response Error: Code=INVALID_SCHEMA_REPLY Message=INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log
FATA[0009] ❌ INVALID_SCHEMA_REPLY SERVER events_api#/methods/read_activity_log

root@~/noobaa5.10# ./noobaa version
INFO[0000] CLI version: 5.10.1
INFO[0000] noobaa-image: noobaa/noobaa-core:5.10.1
INFO[0000] operator-image: noobaa/noobaa-operator:5.10.1

So, if I could summarize, I have a noobaa server (operator) of version 5.8.0 and I had tried CLI version's 5.8.0 ( to match server version) and 5.10.1. But I still receive the same error. As far as I understood, the cli and operator are same binaries and so do you think I should try updating the noobaa server (operator) to latest version?

Regards,

Test: ➜ noobaa-operator git:(6f571fd) ✗ ./build/_output/bin/noobaa-operator-local -n noobaa api events_api export_activity_log '{}' INFO[0000] ✅ Exists: NooBaa "noobaa" INFO[0000] ✅ Exists: Service "noobaa-mgmt" INFO[0000] ✅ Exists: Secret "noobaa-operator" INFO[0000] ✅ Exists: Secret "noobaa-admin" INFO[0000] ✈️ RPC: events.export_activity_log() Request: map[] WARN[0000] RPC: GetConnection creating connection to wss://localhost:63468/rpc/ 0xc000b9fc20 INFO[0000] RPC: Connecting websocket (0xc000b9fc20) &{RPC:0xc0003659a0 Address:wss://localhost:63468/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>} INFO[0000] RPC: Connected websocket (0xc000b9fc20) &{RPC:0xc0003659a0 Address:wss://localhost:63468/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>} ERRO[0001] RPC: HandleResponse for OP res API raw bytes {"op":"res","reqid":"wss://localhost:63468/rpc/-0","took":898.633090000134,"reply":{"out_path":"/public/audit.csv"}} INFO[0001] ✅ RPC: events.export_activity_log() Response OK: took 898.6ms out_path: /public/audit.csv Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

github-actions · 2024-06-04T11:00:58Z

This issue had no activity for too long - it will now be labeled stale. Update it to prevent it from getting closed.

baum mentioned this issue Oct 4, 2022

🧹 Fix event api export_activity_log reply schema #7087

Merged

github-actions bot added the Stale label Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER #7071

Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER #7071

k-e-r-n-e-l-p-a-n-i-c commented Sep 16, 2022

k-e-r-n-e-l-p-a-n-i-c commented Sep 24, 2022

baum commented Oct 2, 2022 •

edited

k-e-r-n-e-l-p-a-n-i-c commented Oct 4, 2022 •

edited

baum commented Oct 6, 2022

k-e-r-n-e-l-p-a-n-i-c commented Oct 6, 2022 •

edited

github-actions bot commented Jun 4, 2024

Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER #7071

Noobaa events_api fails with error INVALID_SCHEMA_REPLY SERVER #7071

Comments

k-e-r-n-e-l-p-a-n-i-c commented Sep 16, 2022

Environment info

Actual behavior

Expected behavior

Steps to reproduce

More information - Screenshots / Logs / Other output

k-e-r-n-e-l-p-a-n-i-c commented Sep 24, 2022

baum commented Oct 2, 2022 • edited

v5.10.1

Latest master

k-e-r-n-e-l-p-a-n-i-c commented Oct 4, 2022 • edited

baum commented Oct 6, 2022

k-e-r-n-e-l-p-a-n-i-c commented Oct 6, 2022 • edited

github-actions bot commented Jun 4, 2024

baum commented Oct 2, 2022 •

edited

k-e-r-n-e-l-p-a-n-i-c commented Oct 4, 2022 •

edited

k-e-r-n-e-l-p-a-n-i-c commented Oct 6, 2022 •

edited