Bump Cli-testing to 1.3.0, update kafka libraries by SylvainSenechal · Pull Request #2341 · scality/Zenko

SylvainSenechal · 2026-03-02T16:23:10Z

Bump cli testing to 1.3.0 => Breaking changes around the keycloak seeder, so updated a few things related to that
Updated node runner to 24
Updated cucumber version from 10.x to 12.x
Updated kubernetes client from 0.2 to 1.4...
Kafka libraries : align all kafka interactions on the new JS native kafka library : platformatic/kafka => drop kafkajs, drop node-rdkafka (and drop the annoying node-gyp that was needed for node-rdkafka, which was always causing weird python warnings when installing). => This new kafka library is compatible with Bunm unlike the previous ones, so later we can investigate if using Bun is worth it (I believe it is 🌚 ), before that we could'nt even try because of this lib

Ended up adding some extra stuff as I was testing my changes in Codespace and found a few stuff bothering me. I think the pr is still not too hard to review.

Can review commit by commit too

bert-e · 2026-03-02T16:23:14Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

Issue: Zenko-5203

SylvainSenechal · 2026-03-11T15:12:21Z

tests/ctst/package.json

    "build": "tsc --build tsconfig.json",
    "lint": "eslint ."
+  },
+  "resolutions": {


related to issues newer azure libraries : https://scality.atlassian.net/browse/ZENKO-5213

SylvainSenechal · 2026-03-11T15:15:02Z

tests/ctst/steps/utils/utils.ts

This was an infinite loop, although catched by global test timeout, still nicer to properly control while loops

SylvainSenechal · 2026-03-11T15:15:46Z

tests/ctst/steps/utils/kubernetes.ts

These are all small syntax changes because I did a huge bump of the kubernetes client from version 0.x to 1.4.0

SylvainSenechal · 2026-03-11T15:18:04Z

tests/ctst/steps/utils/kafka.ts

+ * new `putBucketNotificationConfiguration` to the connector before the
+ * test proceeds to trigger events.
+ */ 
+export async function waitForBucketInConnectorPipeline(


Used in bucket notification : Before, we were doing

putBucketNotificationConfiguration

sleep for 10sec and pray the oplog has picked up the change and updated kafka connector.

It was risky, because triggering the notification event before kafka connector was setup would imply the notification would never be sent

Now we detect the kafka connector is really setup.

I'm kinda sad though : I thought this would fix bucket notif flaky, but it doesn't, so there are other things to check

SylvainSenechal · 2026-03-11T15:19:18Z

tests/ctst/steps/bucket-policies/common.ts

            assert.strictEqual(this.getResult().err?.includes('AccessDenied') ||
-                this.getResult().err?.includes('403'), true);
+                this.getResult().err?.includes('403')||
+                this.getResult().statusCode === 403, true);


For some reason, I got issues with this part not catching the 403 anymore.
Not fully sure why, but with the bump of the cli testing, the version of the sdk has increased quite a lot, from ~3.900 to ~3.100, maybe something changed I don't know

SylvainSenechal · 2026-03-11T15:20:41Z

tests/ctst/features/azureArchive.feature

-    Given that lifecycle is "resumed" for the "e2e-azure-archive" location
-    Then object "obj-1" should be "transitioning" and have the storage class "e2e-azure-archive"
-    And object "obj-2" should be "transitioning" and have the storage class "e2e-azure-archive"
+# This test is flaky, and doesn't make much sense as it is :


@francoisferrand Let's discuss this. I only commented for now, maybe we wanna keep it ?

But looking at the test it doesnt make much sense (cf the comment I added in the code).
But at the same time I feel like I'm missing something, there must be a reason why this test was created

Don't forget to resolve this before merging 😉

SylvainSenechal · 2026-03-11T15:21:40Z

tests/ctst/common/hooks.ts

    cleanupAccount,
 } from './utils';

+import 'cli-testing/hooks/KeycloakSetup';


We have BeforeAll in Cli-testing, no import => they are not run
Imo should be in this repo but its a story for another day

SylvainSenechal · 2026-03-11T15:22:13Z

.github/scripts/end2end/run-e2e-ctst.sh

 POD_NAME="${ZENKO_NAME}-ctst-tests"
 CTST_VERSION=$(sed 's/.*"cli-testing": ".*#\(.*\)".*/\1/;t;d' ../../../tests/ctst/package.json)

-# Configure keycloak


Seed keycloak is now done in a before all from Cli-testing

SylvainSenechal · 2026-03-11T15:25:23Z

tests/ctst/common/common.ts

 import { safeJsonParse } from './utils';
 import assert from 'assert';
-import { Admin, Kafka } from 'kafkajs';
+import { Admin } from '@platformatic/kafka';


Switched the kafka library
Before we were using kafkajs, and node-rdkafka.

These internally relied on librdkafka, a C library. Because of that, we needed to setup quite a few extra stuff to make them work : node-gyp (which was throwing weird python errors when doing yarn install in the Codespace), and also need a bunch of things to be installed in the dockerfile (you'll see later in the pr, librd-XXX).
Now this library is pure js, it removes some dependency, and also it will be compatible with Bun if we wanna use it later

notes:

such comments would be better in the PR comment, so reviewers know before even reading the code

would be good to keep such change in a separate PR (even if it means stacking multiple PRs) - not saying to change it now (too late), but as a goal for next changes

Thanks for the kafka change!

SylvainSenechal · 2026-03-11T15:26:42Z

tests/ctst/steps/azureArchive.ts

        const tarName = await isObjectRehydrated(this, objectName);
        assert(tarName);
-        await AzureHelper.sendBlobCreatedEventToQueue(
+        const sent = await AzureHelper.sendBlobCreatedEventToQueue(


Lost crazy amount of time on this :
sendBlobCreatedEventToQueue was erroring, the error wasn't thrown but instead the function return a boolean that was never checked -_-

francoisferrand · 2026-03-13T10:11:04Z

tests/ctst/common/common.ts

+            const low = earliestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);
+            const high = latestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);


partitions are lists, right? So searching is not efficient...

are partitions "ordered" predictably?

do earliest and latest have the same order?

...could we not just do a "zip" merge, something like:

for (let i = 0; i < partitionCount; i++) { partitions.push({ low: earliestResult[0]?.partitions[I], high: latestResult[0]?.partitions[I] });

I think we don't have guarantee about the order, but the list should be small enough that in practice this find is not too impactfull

tests/ctst/steps/notifications.ts

francoisferrand · 2026-03-13T10:49:54Z

tests/ctst/steps/notifications.ts

+                        offset: msg.offset?.toString(),
+                        value: msg.value,
+                    });
+                    try {


nested try blocks (3 here!) are kind of hard to read, should we not introduce helper functions?

francoisferrand · 2026-03-13T10:52:40Z

tests/ctst/steps/notifications.ts

-                    const eventTypeMatches = notificationType === notification?.eventName;
-                    if (bucketNameMatches && objectNameMatches && eventTypeMatches) {
-                        return true;
+            try {


instead of integrating the lib here, should we modify KafkaHelper directly in cli-testing to change the kafka lib?
or at least we need to remove that code (and dependency) from cli-testing?

I removed the kafka helper from cli-testing, it was only used once in Zenko, no need for such a helper, and this way, no more node-rdkafka in cli testing

francoisferrand · 2026-03-13T10:58:09Z

tests/ctst/cucumber.config.cjs

        format: [
-            'progress-bar',
-            '@cucumber/pretty-formatter',
+            'pretty',


why are we removing @cucumber/pretty-formatter ?
seems a good option to auto-format .freature files?

Pretty formatter is archived (https://github.com/cucumber/cucumber-js-pretty-formatter?tab=readme-ov-file), It's integrated directly in cucumber now (thats why i removed pretty formatter in cli-testing), and pretty formatter is now just "pretty" : https://github.com/cucumber/cucumber-js/blob/main/docs/formatters.md

bert-e · 2026-03-13T12:47:01Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Issue: ZENKO-5203

…file Issue: ZENKO-5203

Issue: ZENKO-5203

maeldonn

LGTM

maeldonn · 2026-03-13T14:30:01Z

tests/ctst/Dockerfile

 FROM ghcr.io/scality/zenko-drctl:$DRCTL_TAG AS drctl

-FROM node:22.19.0-bookworm-slim
+FROM node:24-bookworm-slim


Shouldn't we pin the version like before ?

yeah gonna use 24.14

maeldonn · 2026-03-13T14:31:38Z

tests/ctst/features/azureArchive.feature

-    Given that lifecycle is "resumed" for the "e2e-azure-archive" location
-    Then object "obj-1" should be "transitioning" and have the storage class "e2e-azure-archive"
-    And object "obj-2" should be "transitioning" and have the storage class "e2e-azure-archive"
+# This test is flaky, and doesn't make much sense as it is :


Don't forget to resolve this before merging 😉

maeldonn · 2026-03-13T14:42:00Z

tests/ctst/steps/bucket-policies/common.ts

-            // S3 is correct.
            assert.strictEqual(this.getResult().err?.includes('AccessDenied') ||
-                this.getResult().err?.includes('403'), true);
+                this.getResult().err?.includes('403')||


Do we really need to keep this condition ?

I am not 100% sure anymore tbh, might have been something that failed while i was working, but probably only need one of the two now.
But a/b testing by launching the ci again with and without i too long to just fix this

…2.7.0 Issue: ZENKO-5203

Issue: ZENKO-5203

…ibraries Issue: ZENKO-5203

…he DLQ was ~250 in the tests, stopping sorbetclt from restoring objects Issue: ZENKO-5203

…rs instead of sleep Issue: ZENKO-5203

Issue: ZENKO-5203

…prove debugging Issue: ZENKO-5203

Issue: ZENKO-5203

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch 7 times, most recently from 50ff85e to b441c6e Compare March 4, 2026 09:49

use node 24 for ctst linting step

9f68638

Issue: Zenko-5203

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch 2 times, most recently from b0358e1 to e39e7a2 Compare March 11, 2026 15:10

SylvainSenechal commented Mar 11, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from e39e7a2 to dedf6c3 Compare March 12, 2026 13:05

SylvainSenechal marked this pull request as ready for review March 12, 2026 13:08

SylvainSenechal requested review from a team, benzekrimaha, francoisferrand and maeldonn and removed request for benzekrimaha March 12, 2026 13:08

delthas mentioned this pull request Mar 12, 2026

Add CTST tests for storage usage reporting API #2340

Open

francoisferrand reviewed Mar 13, 2026

View reviewed changes

tests/ctst/steps/notifications.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Mar 13, 2026

View reviewed changes

francoisferrand approved these changes Mar 13, 2026

View reviewed changes

francoisferrand reviewed Mar 13, 2026

View reviewed changes

scality deleted a comment from bert-e Mar 13, 2026

SylvainSenechal added 5 commits March 13, 2026 14:18

use new platformatic kafka lib for ctst

2e26ece

Issue: ZENKO-5203

replace kafkajs library with platformatic kafka

cd7fd01

Issue: ZENKO-5203

remove node gyp following removal of node rd kafka

c37446f

Issue: ZENKO-5203

remove useless tools previously used for node rd kafka in ctst docker…

d7e565a

…file Issue: ZENKO-5203

bump cucumber to 12.7

c728107

Issue: ZENKO-5203

scality deleted a comment from bert-e Mar 13, 2026

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from 4a27e41 to 0c6fedd Compare March 13, 2026 13:24

maeldonn approved these changes Mar 13, 2026

View reviewed changes

SylvainSenechal added 14 commits March 13, 2026 16:46

bump node image to 24 in ctst dockerfile following cucumber bump to 1…

5f3b575

…2.7.0 Issue: ZENKO-5203

bump kubernetes client to 1.4.0

d49a9bb

Issue: ZENKO-5203

bump cli-testing to 1.3.0

c25591a

Issue: ZENKO-5203

seed keycloak from cli testing before all following 1.3.0 bump

61fa240

Issue: ZENKO-5203

update cucumber formatter

9d3b913

Issue: ZENKO-5203

fix infinite loop on verify object location

3403315

Issue: ZENKO-5203

fix 403 auth check

b8ef05f

Issue: ZENKO-5203

add missing cold storage tag to test

b5e165a

Issue: ZENKO-5203

pin azure core client following issues with mismatched azure client l…

70ba790

…ibraries Issue: ZENKO-5203

increase sorbetclt limit to 10000 from default 100, as it was found t…

585d1bd

…he DLQ was ~250 in the tests, stopping sorbetclt from restoring objects Issue: ZENKO-5203

increase notification test reliability by checking for kafka connecto…

f98827e

…rs instead of sleep Issue: ZENKO-5203

remove useless azure archive test

cc4c5e9

Issue: ZENKO-5203

improve uniqueness of object names in tests to avoid collision and im…

37f611f

…prove debugging Issue: ZENKO-5203

add cucumber extension to codespace

879ebee

Issue: ZENKO-5203

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from 0c6fedd to 879ebee Compare March 13, 2026 15:46

		const low = earliestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);
		const high = latestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);

Conversation

SylvainSenechal commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Mar 2, 2026

Hello sylvainsenechal,

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bert-e commented Mar 13, 2026

Waiting for approval

Uh oh!

maeldonn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SylvainSenechal commented Mar 2, 2026 •

edited

Loading

francoisferrand Mar 13, 2026 •

edited

Loading

francoisferrand Mar 13, 2026 •

edited

Loading