Built cloudserver client with Smithy #1

SylvainSenechal · 2025-10-02T14:39:04Z

Related : scality/backbeat#2679
Replacing the "Backbeat client" built with the AWS SDK V2 in Backbeat, with this new client built with Smithy, related to our migration to the AWS SDK V3

ISSUE: CLDSRVCLT-1

LICENCE

SylvainSenechal · 2025-10-02T14:50:33Z

legacy/backbeat-2017-07-01.api.json

I think its nice to have this here as a reference, let me know if you think we should remove it

it is still in (backbeat) git history, should be enough I think.
would be nice however to link to backbeat file on the PR message and/or commit message

👍 agree to delete it

build/smithy/classpath.json

localTests/initBucketForTests.ts

francoisferrand · 2025-10-02T16:22:30Z

build/smithy/source/sources/getMetadata.smithy

should this "build" folder be committed?

looking at package.json, it seems we rebuild them anyway...

We need to have smithy installed to generate the build folder, I feel like it would be more complex if it's not comited ? At what moment would the build generation happen ?
I'm not too sure about this

Should be in the CI to be honest and the pushed with the release

localTests/initBucketForTests.ts

francoisferrand · 2025-10-02T16:30:21Z

service/cloudserver.smithy

+@restJson1
+@sigv4(name: "s3")
+@service(sdkId: "cloudserver")
+service cloudserver {


looking at the bigger picture (i.e. all the APIs supported by cloudserver), this is only one part : i.e. the "Backbeat routes".

eventually (not in this PR ;) we should have other APIs I guess, i.e.

custom listing and other "extensions" to s3 api with extra attributes/... (typically done with a middleware, not sure if/how we want to override or just provide an ready-to-use middleware)

metadata routes https://github.com/scality/cloudserver/blob/development/9.1/lib/routes/routeMetadata.js

backbeat api routes (as used in zenkoclient, i.e. just frontend for backbeat/api extension)

quota API

...

→ given this context, is cloudserver the right name for this specific service?

IMO cloudserver is fine. It's a first iteration so we don't have all of them and I think it's ok. Maybe we should add a TODO.md with which API is missing, the cleanup of the tests (assert of process.env) and the CI? And link them to JIRA ticket

README.md

package.json

build/smithy/classpath.json

francoisferrand · 2025-10-02T16:37:53Z

legacy/backbeat-2017-07-01.api.json

it is still in (backbeat) git history, should be enough I think.
would be nice however to link to backbeat file on the PR message and/or commit message

francoisferrand · 2025-10-02T16:43:08Z

tests/package.json

+  },
+  "dependencies": {
+    "@aws-sdk/client-s3": "^3.896.0",
+    "@scality/cloudserverclient": "file:../build/smithy/source/typescript-codegen"


this works in the same repo (file:..); how will it work from another repo, i.e. with git reference?

depending on the "internal" generation path of the package does not look very good..... could we/should we "wrap" this with a top-level file -maybe manually generated- re-exporting tuff from build/smithy/source/typescript-codegen ? So that other packages could directly use the package.json defined at root directory?

alternatively, we may push the registry... c.f. scality/Guidelines#206
(may be done in a followup PR, adding github actions workflows to build/... everything)

I tried importing it from backbeat
It worked with this :
"@scality/cloudserverclient": "file:../cloudserverclient"
What I did here in the package.json is if you look at "main", "types", "modules", I provided the path to the build folder :
"main": "build/smithy/source/typescript-codegen/dist-cjs"
So when we release a version of this repo, we should be able to use the library this way :
"@scality/cloudserverclient": "github:scality/cloudserverclient#v1.0.0"

Ended up changing it :
We now have a client wrapper setup at the root project, it was needed to be able to provide this client with the error wrapper needed to handle html/xml error parsing

francoisferrand

publishing some overall comments, did not yet review the tests or models yet

README.md

localTests/testApis.ts

README.md

francoisferrand · 2025-10-08T07:30:55Z

build/smithy/source/build-info/smithy-build-info.json

@@ -0,0 +1,244 @@
+{


do we actually need to commit the "build" part?
should it not be dynamically generated?

Discussed. Leave it for now, could be possibly changed later, but needs to setup some more stuff because it would us to be able to install the smithy tool in the pipeline

tests/testMultipleBackend.test.ts

tests/testRaftApis.test.ts

francoisferrand · 2025-10-08T07:37:05Z

tests/testRaftApis.test.ts

+            const raftIdData = await client.send(getRaftIdCommand);
+            console.log('GetRaftId succeeded:', raftIdData);
+            console.log('Raft ID:', raftIdData.RaftId);
+        } catch (err) {


we should not catch the error, as it prevents the test framework from noticing : so the test will always succeed...

(if this is expected, best to just skip the test for now...)

Ideally these tests should be run in a pipeline as functional tests, for now I ran them manually and I think they are fine this way. I have another ticket for setting up a pipeline, although I think this will not be straightforward to test : We need cloudserver to be started with different setup depending on the api we want to test

tests/testSetup.ts

SylvainSenechal · 2025-10-23T07:51:34Z

src/index.ts

+                    });
+                }
+
+                // S3C has an nginx proxy that can return HTML error responses


Logic and comment from https://github.com/scality/backbeat/blob/66c19b092af86f90fe0cf22eb4ce25b6a28cddbf/lib/clients/BackbeatClient.js#L30

francoisferrand · 2025-10-29T16:21:05Z

package.json

+    "JSONStream": "^1.3.5"
+  },
+  "author": "Scality",
+  "license": "SEE LICENSE IN FILE LICENSE",


Should be:

Suggested change

"license": "SEE LICENSE IN FILE LICENSE",

"license": "Apache-2.0",

francoisferrand · 2025-10-29T16:23:03Z

.gitignore


 # Nuxt.js build / generate output
 .nuxt
-dist


Suggested change

dist

# For now, the `dist` folder is commited: so we should not ignore it

#dist

francoisferrand · 2025-10-29T17:21:40Z

tests/testApis.test.ts

@@ -0,0 +1,105 @@
+import {


nit: the test prefix in these files is very redundant ; and the suffix is usually .spec.ts (at least in our code) -> files shoudl be renamed (apis.spec.ts, etc)

As we are in a test folder, we can just use test.apis.ts or test.apisjs.js. spec.ts is used when test file are located next to the src ?

francoisferrand · 2025-10-29T17:23:39Z

tests/testApis.test.ts

+            const data = await client.send(command);
+            console.log('PutData succeeded:', data.Location);
+        } catch (err: any) {
+            console.log('PutData failed:', err);


since you catch the error, the test will actually never fail...
--> the whole try/catch is actually not needed, the test framework will autoamtically catch (and probably print) the error...

francoisferrand · 2025-10-29T17:23:57Z

tests/testApis.test.ts

+
+            const command = new PutDataCommand(putInput as any);
+            const data = await client.send(command);
+            console.log('PutData succeeded:', data.Location);


do we need the log now that we have a test framework?

maybe we want to assert on the value of data.Location ?

francoisferrand · 2025-10-29T17:29:09Z

src/index.ts

+    }
+
+    private createCustomErrorMiddleware = () => {
+        return (next: any) => async (args: any) => {


it would be worth extracting this middleware from CloudserverClient for clarity, something like XmlErrorMiddleware

francoisferrand · 2025-10-29T17:31:53Z

service/cloudserver.smithy

+@service(sdkId: "cloudserver")
+service cloudserver {
+    version: "2017-07-01",
+    operations: [


is the order here relevant/needed for smithy?
if possible, it may be best to keep the list sorted (alphabetically), to make it easier to see that every route is there and avoid "reordering" later?

francoisferrand · 2025-10-29T17:35:45Z

models/putMetadata.smithy

+$version: "2.0"
+namespace cloudserver.client
+
+@idempotent


not sure this API is really idempotent : esp. if the versoinId is not specified (but the bucket is versionned), this would create a new version on every call?

francoisferrand · 2025-10-29T17:39:37Z

models/putdata.smithy

+@streaming
+blob StreamingBlob
+
+@idempotent


should this be considered idempotent?

every call with write new data (in the backend), and update the specified Object version to 'point' to that data:

from a user's perspective, should be idempotent indeed (as long as the content of payload is also the same)

from the server's perspective, data would however get written repeatedly, and the (internal) "key" of the data would get updated. (hopefully, the previous data would not be lost -creating orphan-, but not sure...)

do you know how this should be considered & what is the (practical) impact of this trait?

francoisferrand · 2025-10-29T17:42:30Z

models/multipleBackendPutObject.smithy

+    @httpHeader("X-Scal-Request-Uids")
+    RequestUids: String,
+    @httpPayload
+    @default("")


is there really a need for such default? (or is it a side-effect of smithy implementation, where an "empty" body is just skipped and we don't get anything without default?)

francoisferrand · 2025-10-29T17:43:16Z

models/multipleBackendPutObject.smithy

+@streaming
+blob StreamingBlob
+
+@idempotent


may not be idempotent, esp. in case bucket is versionned and X-Scal-Version-Id is not set

francoisferrand · 2025-10-29T17:46:49Z

models/multipleBackendPutObject.smithy

+
+list LocationMDList {
+    member: LocationMDObj
+}
+
+structure LocationMDObj {
+    /// Storage key for this location
+    key: String,
+    /// Size of the data stored at this location
+    size: Integer,
+    /// Start position/offset for this data segment
+    start: Integer,
+    /// Name of the data store where this is located
+    dataStoreName: String,
+    /// Type of the data store (e.g., file, mem, etc.)
+    dataStoreType: String,
+    /// ETag from the data store for this location
+    dataStoreETag: String,
+    /// Version ID in the data store for this location
+    dataStoreVersionId: String
+}


nit: these are shared types, can they be moved to a separate smithy file? (locationMD.smithy)

francoisferrand · 2025-10-29T17:48:14Z

models/multipleBackendPutMPUPart.smithy

+blob StreamingBlob
+
+/// Uploads a part for a multipart upload to multiple backend storage
+@idempotent


does this not create a new part on every call, making it non-idempotent?

francoisferrand · 2025-10-29T17:50:45Z

models/multipleBackendPutObject.smithy

+    @required
+    @httpLabel
+    Bucket: String,
+    @required


nit: skip line between fields (like in other files), makes it much easier to read

francoisferrand · 2025-10-29T17:54:11Z

models/multipleBackendCompleteMPU.smithy

+
+    @httpHeader("X-Scal-Tags")
+    Tags: String,
+    @httpHeader("X-Scal-Request-Uids")


Suggested change

@httpHeader("X-Scal-Request-Uids")

@httpHeader("X-Scal-Request-Uids")

francoisferrand · 2025-10-29T17:54:53Z

models/multipleBackendAbortMPU.smithy

+
+    @httpHeader("X-Scal-Upload-Id")
+    UploadId: String,
+    @httpHeader("X-Scal-Request-Uids")


Suggested change

@httpHeader("X-Scal-Request-Uids")

@httpHeader("X-Scal-Request-Uids")

francoisferrand · 2025-10-29T17:55:45Z

models/listLifecycleOrphans.smithy

+    @httpLabel
+    @required
+    Bucket: String,
+


extra space at EOL (here and other places)

francoisferrand · 2025-10-29T17:56:26Z

models/listLifecycleOrphans.smithy

+    /// Limits the response to keys that begin with the specified prefix
+    @httpQuery("prefix")
+    Prefix: String,
+    @httpHeader("X-Scal-Request-Uids")


Suggested change

@httpHeader("X-Scal-Request-Uids")

@httpHeader("X-Scal-Request-Uids")

francoisferrand · 2025-10-29T17:56:48Z

models/listLifecycleNonCurrents.smithy

+    /// Limits the response to keys that begin with the specified prefix
+    @httpQuery("prefix")
+    Prefix: String,
+    @httpHeader("X-Scal-Request-Uids")


Suggested change

@httpHeader("X-Scal-Request-Uids")

@httpHeader("X-Scal-Request-Uids")

(in most files)

francoisferrand · 2025-10-29T18:02:18Z

models/getObjectList.smithy

+    @required
+    @httpLabel
+    Bucket: String,
+    @httpHeader("X-Scal-Request-Uids")


Suggested change

@httpHeader("X-Scal-Request-Uids")

@httpHeader("X-Scal-Request-Uids")

francoisferrand · 2025-10-29T18:03:46Z

models/getObject.smithy

+    output: GetObjectOutput,
+}
+
+structure GetObjectInput {


is this not the same type used in AWS ?
could we import those types and inherit from them (for custom/extra fields we support)?

something like:

structure GetObjectInput extends aws.GetObjectInput { @httpHeader("X-Scal-Request-Uids") RequestUids: String }

models/deleteObjectFromExpiration.smithy

francoisferrand · 2025-10-29T18:06:57Z

models/deleteBucketIndexes.smithy

+$version: "2.0"
+namespace cloudserver.client
+
+@http(method: "POST", uri: "/_/backbeat/index/{Bucket}?operation=delete")


don't we need this:

@idempotent @suppress(["HttpMethodSemantics.UnexpectedPayload"])

(it has a body in the definition, and delete should be idempotent...)
(or maybe there should not be a body?)

francoisferrand · 2025-10-29T18:09:30Z

tsconfig.json

+    "target": "es2020",
+    "module": "commonjs",
+    "lib": ["es2020"],
+    "outDir": "./dist",


nit: should we output to ./buid instead, so all the generated code is in a single place?

francoisferrand · 2025-10-29T18:19:16Z

LGTM, some cleanup (extra space / missing line in models...) to do.

Not sure what is the effect of @idempotent, but in some case this seems unclear if appropriate: so we may want to dig a bit more

DarkIsDude · 2025-10-31T14:21:08Z

tests/testApis.test.ts

@@ -0,0 +1,105 @@
+import {


As we are in a test folder, we can just use test.apis.ts or test.apisjs.js. spec.ts is used when test file are located next to the src ?

DarkIsDude · 2025-10-31T14:23:26Z

service/cloudserver.smithy

+@restJson1
+@sigv4(name: "s3")
+@service(sdkId: "cloudserver")
+service cloudserver {


IMO cloudserver is fine. It's a first iteration so we don't have all of them and I think it's ok. Maybe we should add a TODO.md with which API is missing, the cleanup of the tests (assert of process.env) and the CI? And link them to JIRA ticket

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch from 920e18a to 356c165 Compare October 2, 2025 14:46

SylvainSenechal commented Oct 2, 2025

View reviewed changes

LICENCE Show resolved Hide resolved

SylvainSenechal commented Oct 2, 2025

View reviewed changes

build/smithy/classpath.json Show resolved Hide resolved

SylvainSenechal commented Oct 2, 2025

View reviewed changes

localTests/initBucketForTests.ts Outdated Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch from 8101fc0 to 103d2ed Compare October 2, 2025 15:22

francoisferrand reviewed Oct 2, 2025

View reviewed changes

README.md Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch 3 times, most recently from c8442a6 to c6c998f Compare October 2, 2025 20:27

francoisferrand reviewed Oct 3, 2025

View reviewed changes

localTests/testApis.ts Outdated Show resolved Hide resolved

BourgoisMickael reviewed Oct 3, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

BourgoisMickael reviewed Oct 3, 2025

View reviewed changes

README.md Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch 3 times, most recently from 766fbca to 125bc0d Compare October 6, 2025 12:49

francoisferrand reviewed Oct 8, 2025

View reviewed changes

tests/testMultipleBackend.test.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Oct 8, 2025

View reviewed changes

tests/testRaftApis.test.ts Show resolved Hide resolved

francoisferrand reviewed Oct 8, 2025

View reviewed changes

tests/testSetup.ts Outdated Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch 5 times, most recently from 0e5f323 to dfc7d0f Compare October 23, 2025 07:48

SylvainSenechal commented Oct 23, 2025

View reviewed changes

fixup licence

e51d830

francoisferrand reviewed Oct 29, 2025

View reviewed changes

models/deleteObjectFromExpiration.smithy Show resolved Hide resolved

francoisferrand reviewed Oct 29, 2025

View reviewed changes

SylvainSenechal requested review from a team, DarkIsDude and francoisferrand October 31, 2025 09:37

DarkIsDude approved these changes Oct 31, 2025

View reviewed changes

	"license": "SEE LICENSE IN FILE LICENSE",
	"license": "Apache-2.0",

	dist
	# For now, the `dist` folder is commited: so we should not ignore it
	#dist

	@httpHeader("X-Scal-Request-Uids")

	@httpHeader("X-Scal-Request-Uids")

Built cloudserver client with Smithy #1

Are you sure you want to change the base?

Built cloudserver client with Smithy #1

Uh oh!

Conversation

SylvainSenechal commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

francoisferrand Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

francoisferrand Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal commented Oct 2, 2025 •

edited

Loading

francoisferrand Oct 2, 2025 •

edited

Loading

francoisferrand Oct 2, 2025 •

edited

Loading

francoisferrand Oct 2, 2025 •

edited

Loading

francoisferrand Oct 29, 2025 •

edited

Loading

francoisferrand Oct 29, 2025 •

edited

Loading