Skip to content

Vyomi v2.0.9

Choose a tag to compare

View all tags
@github-actions github-actions released this 21 Jun 22:38
· 5 commits to main since this release
v2.0.9
c77286f

Cross-cloud native-SDK conformance extended to Secrets + KMS — and a transport-conformance map that fixes how every Messaging / Secrets / KMS surface is reached, native-SDK-as-is, across AWS / GCP / Azure. Following the principle that a green conformance test must give a developer full confidence to build a real app with the unmodified vendor SDK (the backing tech — ElasticMQ, Vault, Azurite, emulators — is our private pick, never something their code depends on), we mapped all nine surfaces by the transport the native SDK actually speaks and wired only the ones that pass that bar, flagging the rest honestly rather than faking a pass.

Added

  • /secret/{cloud} + /kms/{cloud} conformance flows in samples/java/cloud-probe. New CloudProbe.probeSecret() / probeKms() lifecycle methods (create → add-version → access + verify → delete for secrets; ensure-key → encrypt → decrypt → verify for KMS), surfaced as two new ProbeController endpoints alongside /probe, /object, /item.
  • GCP Secret Manager + Cloud KMS over the SDK's native REST (HttpJson) transport. GCP ships no emulator for these (unlike Pub/Sub / Firestore) and the default transport is gRPC to *.googleapis.com, which the appliance can't serve — so the native SDK as-is wouldn't connect. The google-cloud-secretmanager / google-cloud-kms Java SDKs expose newHttpJsonBuilder(), a REST transport we point at the appliance (caddy HTTPS endpoint, NoCredentials) — still the native SDK, configured the way it supports (like AWS endpointOverride). New pom deps (google-cloud-secretmanager, google-cloud-kms), ProbeEnv.gcpRestEndpoint() / gcpLocation(), and the GcpProbe HttpJson clients. KMS key-ring/key creation is best-effort (the Vault-transit backing may auto-provision on first encrypt) so the encrypt/decrypt round-trip — the real assertion — always runs.
  • Azure Storage Queue server handler — the AMQP→HTTP substitution for Service Bus. Azure's native messaging SDK speaks AMQP (unserveable) and the MS Service Bus emulator is non-viable, so Azure messaging conformance is delivered through the native azure-storage-queue HTTP SDK instead. New core/azure_queue_store.py (mirrors azure_blob_store.py) backs queues with Azurite's queue service — docker-compose.appliance.yml now runs the full azurite (blob :10000 + queue :10001) with CLOUDLEARN_AZURITE_QUEUE_URL, and azure-storage-queue is added to requirements.txt. core/azure_dataplane.py serves the native Queue REST surface at /azure-data/queue/{account} (a dispatch passthrough, so it never collides with the blob handler that shares the x-ms-version signature): it terminates the protocol — create/delete queue, enqueue/dequeue/delete-message, list — and re-issues via the Python SDK to Azurite, mapping each (space, account, queue) onto an Azurite queue keyed by hash with the simulator names in metadata. So an unmodified azure-storage-queue client round-trips create→send→receive→delete.
  • All nine Messaging / Secrets / KMS flows now implemented across AWS / GCP / Azure (compiles clean). AwsProbe — SQS (create/send/receive/ack/delete), Secrets Manager (create/get/delete), KMS (createKey/encrypt/decrypt/scheduleDelete), all HTTP+SigV4. GcpProbe — Pub/Sub via the gRPC emulator (topic+subscription/publish/pull/ack), Secret Manager + KMS via HttpJson. AzureProbe — Storage Queue (the AMQP→HTTP substitution for Service Bus, Azurite conn-string), Key Vault Secrets, Key Vault Keys (RSA encrypt/decrypt via CryptographyClient) — Key Vault clients use a fake TokenCredential (the appliance ignores auth) against the caddy HTTPS endpoint. New pom deps (sqs/secretsmanager/kms; google-cloud-pubsub; azure-storage-queue/azure-security-keyvault-secrets/azure-security-keyvault-keys) + ProbeEnv helpers (pubsubEmulatorHost, azureQueueConnectionString, azureKeyVaultEndpoint, caddyHost).

Decided (research-validated)

  • Transport-conformance map — 6 of 9 surfaces are natively reachable as-is. Native-SDK pass: AWS SQS (ElasticMQ, SigV4), Secrets Manager, KMS; GCP Pub/Sub (gRPC emulator); Azure Key Vault Secrets + Keys (REST). Three were real gaps: GCP Secret Manager + KMS (gRPC, no emulator) — now closed via the HttpJson transport above; and Azure Service Bus (see below).
  • Azure Service Bus emulator rejected — Azure messaging will use native azure-storage-queue on Azurite. Embedding Microsoft's Service Bus emulator fails three independent blockers: (1) runtime entity create/delete via the Administration client is .NET-only, so the native Java SDK can't create/delete against it (management runs over a non-TLS custom port non-.NET SDKs won't honor) — fatal for a create→send→receive→delete probe; (2) it requires an amd64-only MS SQL Server 2022 sidecar (the ARM-capable Azure SQL Edge was retired 2025-09-30); (3) ~1.6 GB SQL image + 2 GB RAM exceeds the WebVM disk budget and ships under a dev/test-only, non-redistributable EULA. The pragmatic native-SDK path is azure-storage-queue over HTTP, backed by the Azurite container already used for Azure Blob — a different service, stated plainly, but a real native Azure queue SDK with full runtime CRUD.

Added (console)

  • GCP console CRUD — Pub/Sub, Secret Manager, Cloud KMS (conformant). Catalog-driven like AWS. Pub/Sub was already conformant (the console handlers route to the gRPC emulator via core/gcp_pubsub_emulator when available — the same backend the native SDK reads). Secret Manager + Cloud KMS repointed off the in-process /api/gcp/extras/* onto new Vault-backed routes in core/vault_routes._register_gcp: /api/gcp/secrets (list/create/delete via vc.kv_*("gcp",…) + a __idx__secrets index, also maintained by the native gcp_secrets_create/addVersion) and /api/gcp/kms/keys (list from a __idx__kms index the native gcp_kms_encrypt populates; create via vc.transit_create_key). Also added the previously-missing native DELETE /v1/projects/{project}/secrets/{secret} (so the google-cloud-secretmanager SDK's deleteSecret + console delete land). providers/gcp_catalog.py repointed. Validated: py_compile.
  • AWS console CRUD — SQS, Secrets Manager, KMS (conformant). The AWS console's list/create/delete is catalog-driven, so the fix is backend + catalog wiring (no new UI). SQS was already conformant (catalog /api/sqs/queues shares the in-memory store with the JSON-RPC native path). Secrets Manager + KMS were repointed off the in-process /api/aws/extras/* (a false green — invisible to the Vault-backed native SDKs) onto new Vault-backed routes in core/vault_routes.py: /api/aws/secrets (list/create/delete via vc.kv_*, with a KV index also maintained by the native _aws_secrets_dispatch so SDK-created secrets appear) and /api/aws/kms/keys (list from the _kms_keys index the native CreateKey populates; create via vc.transit_create_key). providers/aws_catalog.py now points both services at these. So a console-created secret/key is visible to the native AWS SDK and vice-versa. Validated: py_compile.
  • Azure Key Vault data plane completed — real RSA keys + secrets list/delete. The native CryptographyClient does RSA-OAEP client-side (fetches the public key, encrypts locally, calls the service only to decrypt), which Vault transit's symmetric encrypt/decrypt can't satisfy — so core/vault_routes.py now generates a real RSA-2048 keypair (via cryptography, private key persisted in Vault KV) and serves the full Key Vault keys surface: create (returns a public JWK with n/e), get/get-version, list, delete, and RSA-OAEP encrypt/decrypt (SHA-1 + SHA-256). The kid is built from the request Host so the SDK sends decrypt back to the appliance, not real *.vault.azure.net. Secrets gain list + delete (with a KV-backed name index). This closes the gap that previously blocked the secrets/keys console and AzureProbe.probeKms's createKey/round-trip.
  • Azure console CRUD — Queues, Key Vault Secrets, Key Vault Keys (UI + routes, conformant). Three new static/azure-console.html sub-blade renderers wired to the same backends the native SDKs use, so console writes are SDK-visible and vice-versa (never the ARM _state, which would be a false green): Queues (storagequeues, under Storage accounts) → /api/azure/storage/{account}/queues (new route in core/azure_dataplane.py) → Azurite via azure_queue_store; Secrets (kvsecrets) + Keys (kvkeys, under Key Vault) → the Vault-backed /azure-data/keyvault/{vault}/... data plane. The Storage "Queues" sub-blade was promoted from stub, and the Key Vault Secrets/Keys blades were repointed from the ARM-state listChildResources stubs to the conformant data plane (core/azure_subblades.py). Validated: py_compile + node --check on the console JS.
  • Full secret/key edit (the U in CRUD) across all three consoles — modeled on each real provider's actual console gesture, not a generic dialog. Secrets were create/read/delete only; KMS keys had no edit at all. Secret value edit — a detail-view "Secret value" tab with Retrieve → (Key/value table or Plaintext) → Edit → Save, PUTing the resource (AWS secretvalue + GCP renderSecretValueTab; Azure inline reveal+edit in kvsecrets) — mirrors the AWS console's in-place edit rather than re-running the create wizard. KMS metadata edit — each provider exposes what its console actually edits (real KMS consoles have no encrypt/decrypt UI): AWS description + Enable/Disable (kmsconfig tab → GET/PUT /api/aws/kms/keys/{id} against _kms_keys); GCP labels + rotation period (kmsconfig "Rotation & labels" tab → GET/PUT /api/gcp/kms/keys/{name}, KV-backed __kmsmeta__ store); Azure Enable/Disable per key (Status column + button → native PATCH {attributes:{enabled}}, the same call KeyClient.updateKeyProperties() makes, backed by a __attrs flag in Vault KV). Validated: py_compile + node --check, and create→edit→read round-trips on a running appliance for all three.

Fixed (live VM conformance run — 2026-06-22)

Verified the whole feature against a running local appliance (vyomi/appliance:2.0.9 built from source; full backend stack incl. Azurite-with-queue, the GCP emulators, Vault, and caddy TLS) by running the cloud-probe harness across all 12 endpoints. Result: 11/12 green — every one of the nine new Messaging / Secrets / KMS flows passes across AWS / GCP / Azure. The runtime issues that surfaced (and the fixes):

  • AWS SQS now answers JSON-protocol clients in JSON. _sqs_json_dispatch (server.py) proxied JSON-protocol SQS requests (AWS SDK v2 / modern boto3, X-Amz-Target) to ElasticMQ, which only speaks the legacy query protocol and returns XML — the proxy translated the request but not the response, so the SDK failed to unmarshall (Unexpected character '<'). JSON clients are now served from the in-memory store (which implements visibility-timeout + FIFO); the legacy query path keeps using ElasticMQ.
  • Azure Key Vault + Storage Queue reachable by the native SDKs (host-based routing). The CryptographyClient and azure-storage-queue SDKs rebuild request URLs from the host only, dropping any path prefix — so a /azure-data/... path endpoint can't work (requests hit /keys/… / /{queue}/… at the root and fell to the S3 catch-all → 404). Real Azure gives each vault/account its own hostname, so we mirror that: a Host→path rewrite middleware (core/middleware.py) maps {vault}.vault.* → /azure-data/keyvault/{vault}/… and {account}.queue.* → /azure-data/queue/{account}/… (runs before the blob rule, since queue traffic also carries x-ms-version); Key Vault id/kid responses are now host-rooted (https://{host}/keys|secrets/…) so the SDK parses them. The probe targets *.vault.localtest.me / *.queue.localtest.me (resolve to 127.0.0.1 — no DNS/hosts edits) with a *.vault.localtest.me mkcert cert.
  • GCP Cloud KMS createKeyRing / createCryptoKey / getCryptoKey no longer 404. The native google-cloud-kms SDK creates the ring + key before encrypting; those endpoints were unserved (vault_routes._register_gcp). Added — encrypt already auto-provisions the transit key, so create just registers it.
  • GCP console Secret Manager / Cloud KMS CRUD now resolves (was {"detail":"Not found"}). The provider_api_alias_middleware rewrites /api/gcp/{X} → /api/{X} for any GCP path not in is_gcp_native_path (core/app_context.py); the new console routes /api/gcp/secrets + /api/gcp/kms/keys weren't allowlisted, so they were stripped to /api/secrets / /api/kms/keys (no route → 404). Added both prefixes to is_gcp_native_path (consistent with every other GCP console service — iam/rds/sqs/storage/…). AWS has no such alias, which is why /api/aws/secrets always worked. All three consoles' new-service CRUD now resolves.
  • Azure Key Vault secret/key responses. Added attributes.recoveryLevel (the SDK NPE'd on a null DeletionRecoveryLevel) and a trailing-slash/version route (/secrets/{secret}/{version:path}, since getSecret requests /secrets/{name}/).
  • Readiness banner no longer stuck at ~52%. The cloudlearn-* → vyomi-* service rename left core/appliance_readiness.py probing the old hostnames (which no longer resolve) → the banner hung in "loading". Functionality was unaffected (backends are reached via the env URLs — only the readiness display); hosts realigned to vyomi-*.
  • caddy preserves the Host port. header_up Host {http.request.hostport} (was {host}, which dropped the port) — so the Key Vault kid carries :9443 and the SDK's decrypt call returns to the appliance instead of :443.
  • (probe) Jackson alignment. samples/java/cloud-probe adds jackson-dataformat-xml (Spring-Boot-managed 2.17.2) — azure-core's XML mapper LinkageError'd (jackson-dataformat-xml=unknown) and broke Key Vault response deserialization.

Status

  • VM-verified: 11/12 conformance endpoints green; all 9 new-service flows pass. The single red — probe/azure — is the pre-existing Azure Cosmos query gap (the SDK asks the gateway for a query plan it doesn't implement), inside the object+NoSQL /probe endpoint and unrelated to this work. The local-run recipe (compose --profile full, mkcert cert + JKS truststore, host/port env, CLOUDLEARN_TIER_ENFORCE=0) is captured in the project notes.

Docs

  • Two Knowledge Center articles — transport-substitution case studies — drafted (portal/marketing/articles/amqp-vs-https.md, grpc-vs-https.md; outline in PLAN-knowledge-center-conformance.md). Each is one focused service where the native SDK's default transport isn't serveable, and how the SDK keeps working as-is over a transport we can serve — carrying the threads tech-choice · trade-offs · assurances · SDK-conformance · what-is-guaranteed. Pending: final copy-edit + canonical-URL wiring + cross-post.
    • "AMQP vs HTTPS" — Azure messaging. Native azure-messaging-servicebus speaks AMQP 1.0 (unserveable; the MS Service Bus emulator is non-viable), so Azure queue conformance is delivered over HTTP via the native azure-storage-queue SDK on Azurite — a different service, stated plainly.
    • "gRPC vs HTTPS" — GCP Secret Manager / Cloud KMS. Native SDKs default to gRPC with no Google emulator, so we drive their own HttpJson (REST/HTTPS) transport against the appliance — still the native SDK, configured the way it supports.

Artifacts

  • SHA256SUMS
  • cloud-learn-0.1.0.tar.gz
  • cloud-learn-2.0.9-1.noarch.rpm
  • cloud-learn-2.0.9.tar.gz
  • cloud-learn_2.0.9_all.deb

Docker image: docker pull vyomi/appliance:2.0.9
Install: curl -fsSL https://raw.githubusercontent.com/vyomi-cloud/appliance/main/install.sh | bash
SHA256 checksums: see SHA256SUMS in attached artifacts.

Assets 7
Loading