AWS parity E2E audit: bugs, gaps & enhancements across all 30 services

[thzgajendra]

AWS parity E2E audit — bugs & enhancements (issue draft)

Method. Ran the standalone binary cloudemu serve --providers aws --aws-port 4566 and drove real aws-sdk-go-v2 clients (BaseEndpoint=http://127.0.0.1:4566) through dependency-ordered lifecycles per resource family — create → describe (with field assertions) → modify/associate → list → tag → sub-resources → delete → error paths. Every non-pass was hand-verified to separate real defects from bad test inputs.

Coverage. All 30 AWS service handlers exercised E2E (~250 operation calls): ec2, s3, iam, sts, dynamodb, lambda, rds, redshift, sqs, sns, eventbridge, secretsmanager, ssm, cloudwatch, cloudwatchlogs, ecr, ecs, eks, elbv2, elasticache, memorydb, keyspaces, route53, networkfirewall, resourcegroupstaggingapi, sagemaker, bedrock, bedrockagent, bedrockagentruntime, resourceexplorer2. Depth = representative dependency-ordered lifecycle per service (not literally every one of each service's SDK operations; EC2 additionally got a full 150-action reachability sweep). Runtime data-planes (model inference) not exercised.
Overall the emulator is strong — the vast majority of core CRUD + error paths pass, and it correctly enforces dependency violations, resource state, and CIDR overlaps.

Parity coverage vs full AWS API (implemented ops ÷ total SDK ops)

High-level breadth (denominator = the entire AWS API for the service, incl. many niche ops an emulator need not cover, so low % is expected for large services):

Service	impl/SDK	Service	impl/SDK	Service	impl/SDK
ec2	150/773	rds	79/168	sagemaker	113/403
iam	39/176	ecs	37/77	memorydb	33/45
keyspaces	18/19	elbv2	18/51	dynamodb	15/57
eventbridge	14/57	cloudwatch	10/46	elasticache	10/75
sns	9/42	redshift	9/141	cloudwatchlogs	9/118
ssm	11/152	sqs	7/23	secretsmanager	7/23
ecr	7/58	networkfirewall	20/84	resourcegroupstaggingapi	9/9
s3 / lambda / route53 / eks / bedrock*	path/target-dispatch (subset)

Thinnest relative to common usage: redshift, cloudwatchlogs, ssm, elasticache, ecr, resourceexplorer2.

---

Cross-cutting themes (fix these first — highest leverage)

A. Tagging is systemically missing/inconsistent

The tag operations users rely on for every IaC tool are absent across many services:

• ec2 CreateTags / DeleteTags — InvalidAction
• dynamodb TagResource — UnknownOperationException
• sns TagResource — InvalidAction
• secretsmanager TagResource — UnknownOperationException
• ssm AddTagsToResource — UnknownOperationException
• sqs TagQueue — UnknownOperationException
• lambda TagResource — 405 + non-JSON body (SDK can't deserialize → wire bug)

Impact: Terraform/CloudFormation/CDK tag nearly everything; tag-on-create or post-create tagging fails. A shared tagging surface would close most of these at once.

B. Set*/Modify* attribute operations commonly missing

• sqs GetQueueAttributes / SetQueueAttributes — High: GetQueueAttributes is how you read a queue ARN (event-source mappings, DLQ wiring, S3→SQS notifications). Its absence breaks many real flows.
• sqs PurgeQueue
• sns SetTopicAttributes
• elasticache ModifyCacheCluster
• redshift CreateClusterParameterGroup
• cloudwatchlogs PutRetentionPolicy
• cloudwatch SetAlarmState
• secretsmanager UpdateSecret — High: updating a secret's metadata is a routine op

D. Unimplemented REST/JSON paths return a 405 HTML page (breaks SDK error handling) — Med

Some unimplemented operations on REST/JSON services return an HTTP 405 + an HTML error body instead of a structured JSON API error. The SDK then fails to deserialize the response (invalid character '<') rather than surfacing a clean "operation not supported". Seen on: lambda TagResource, resourceexplorer2 CreateIndex / GetDefaultView. Users get an opaque parse error instead of an actionable API error.

C. EC2 Describe* by nonexistent ID returns empty success (systemic) — High

Describe{Vpcs,Instances,Volumes,SecurityGroups} with an explicit missing ID returns an empty set with no error, where real AWS returns Invalid*.NotFound. Confirmed across 4 resource types → almost certainly the shared EC2 ID-filter path. Breaks existence checks, Terraform refresh/drift, and "wait-until-deleted" polls. One shared fix likely covers all EC2 types.

---

Per-service findings

Service	Operation(s)	Type	Sev	Note
ec2	Describe{Vpcs,Instances,Volumes,SecurityGroups}(missing id)	BUG	High	no Invalid*.NotFound (theme C)
ec2	CreateTags / DeleteTags	GAP	High	tagging (theme A)
ec2	CreateNetworkInterface	GAP	Med	ENI describe/detach/delete exist; create missing
ec2	MonitorInstances / DescribeInstanceStatus	GAP	Med	common instance ops
ec2	DescribeRegions / DescribeInstanceTypes	GAP	Med	bootstrap/validation calls
s3	HeadBucket (existing & missing)	BUG	Med	HTTP HEAD /bucket verb not routed → 405; breaks bucket-exists check
s3	PutBucketTagging (PUT /bucket?tagging)	BUG	Med	?tagging subresource mis-routed to CreateBucket (BucketAlreadyOwnedByYou)
sqs	GetQueueAttributes / SetQueueAttributes	GAP	High	queue ARN/config (theme B)
sqs	PurgeQueue / TagQueue	GAP	Med
sqs	GetQueueUrl(missing)	ENH	Low	code QueueDoesNotExist vs AWS AWS.SimpleQueueService.NonExistentQueue
sns	SetTopicAttributes / TagResource	GAP	Med
lambda	UpdateFunctionConfiguration / PublishVersion / CreateAlias / AddPermission	GAP	High	404 "unsupported Lambda path" — versioning/alias/permission/config
lambda	TagResource	BUG-WIRE	Med	405 + HTML body → SDK deserialize error
dynamodb	TagResource / DescribeTimeToLive	GAP	Med
iam	PutRolePolicy	GAP	Med	inline role policies
secretsmanager	UpdateSecret / TagResource	GAP	High/Med
ssm	AddTagsToResource	GAP	Med
ecr	GetAuthorizationToken	GAP	High	required for image push/pull auth (docker login)
ecr	SetRepositoryPolicy	GAP	Low
elasticache	ModifyCacheCluster	GAP	Med
redshift	CreateClusterParameterGroup	GAP	Med
cloudwatchlogs	PutRetentionPolicy	GAP	Med
cloudwatch	SetAlarmState	GAP	Low	also returns UnknownError message
cloudwatch	PutMetricData (empty MetricData)	BUG	Low	rejects (correct) but message InvalidParameterValue: UnknownError unhelpful
resourceexplorer2	CreateIndex / GetDefaultView	BUG-WIRE	Med	405 + HTML body → SDK deserialize error (theme D); Search/ListViews/ListIndexes/ListResources work
bedrock / bedrockagent / bedrockagentruntime	list/get/create + Retrieve	—	—	all passed (ListFoundationModels, GetFoundationModel, guardrails, CreateAgent, ListKnowledgeBases, Retrieve)

Verified-correct behavior (NOT bugs — excluded)

• ec2 DeleteSubnet/DeleteVpc → DependencyViolation (I hadn't removed ENI/NAT) ✔ good FK enforcement
• ec2 ModifyInstanceAttribute (type) on a running instance → IncorrectInstanceState ✔ correct
• ec2 CreateVpcPeeringConnection self-peer → CIDR-overlap rejection ✔ correct
• s3 DeleteBucket on non-empty bucket → BucketNotEmpty ✔ correct
• eks DeleteCluster with a live nodegroup → InvalidRequestException ✔ correct

Passed cleanly (solid)

rds (instances/snapshots/clusters/param+subnet groups), ecs (cluster/task-def/service/run-task), sagemaker (model/endpoint-config/endpoint/notebook/training-job), dynamodb data-plane (put/get/query/scan/update/batch), s3 object data-plane, iam roles/users/policies/instance-profiles/groups, route53 zones+record sets, keyspaces, memorydb, eks cluster+nodegroup, elbv2 LB/target-group/listener/register, redshift cluster lifecycle, bedrock list ops, and all non-EC2 missing-resource error paths (proper *NotFound).

[thzgajendra]

Update: user-perspective E2E via the AWS CLI against the live server

Follow-up pass driving the running cloudemu serve with the real aws CLI v2 (--endpoint-url http://localhost:4566, dummy creds) through realistic user workflows across ~18 services. This surfaced issues the SDK harness missed — because the CLI uses different wire protocols than the Go SDK for some services.

🔴 High

1. CloudWatch is fully broken via the AWS CLI. Every op (list-metrics, put-metric-data, describe-alarms) returns InvalidAction: unknown action: <Op>. Root cause: the CLI sends CloudWatch (monitoring) as the query protocol (Content-Type: application/x-www-form-urlencoded, body Action=ListMetrics&Version=2010-08-01), but the cloudemu CloudWatch handler only Matches on Smithy-Protocol: rpc-v2-cbor. So the EC2 handler (which claims form-encoded POSTs) grabs the request and rejects it. SDK/CBOR clients work; all CLI users are broken.
   • Fix idea: disambiguate query-protocol requests by the SigV4 credential scope service in the Authorization header (.../us-east-1/monitoring/aws4_request ⇒ CloudWatch), or have the CloudWatch handler also accept its query protocol.
2. ECR get-login-password fails — GetAuthorizationToken not implemented ⇒ cannot docker login to ECR ⇒ no image push/pull.
3. SQS GetQueueAttributes missing ⇒ cannot read a queue's QueueArn ⇒ cannot wire SNS→SQS, Lambda event-source mappings, or DLQs.
4. Lambda invoke does not execute function code — returns FunctionError: Unhandled, payload {"errorMessage":"no handler registered","errorType":"HandlerError"}.

🟠 Medium

5. SQS CreateQueue returns a real-AWS URL host — https://sqs.us-east-1.amazonaws.com/000000000000/<q> instead of the emulator endpoint. Works via CLI (endpoint override), but a raw SDK caller that reuses the returned QueueUrl without an endpoint override would hit real AWS. (LocalStack returns http://localhost:4566/....)
6. EC2 instance describe omits standard fields users rely on: Placement/AvailabilityZone, VpcId (null even though the vpc-id filter works — inconsistent), Architecture, PrivateDnsName, RootDeviceName, and the auto-assigned default security group.
7. Missing/ malformed ARNs & fields: DynamoDB CreateTable → no TableArn (also ItemCount null); ECR repository → repositoryArn null; EventBridge PutRule → RuleArn malformed arn:aws:events:::rule/default/nightly (no region/account); EKS cluster → version null.
8. S3 ETag is a SHA-256 hex, but real S3 returns the MD5 for simple (non-multipart) uploads ⇒ breaks client-side integrity/ETag comparisons.

🟢 Low

• Secrets Manager secret ARN lacks the real 6-char random suffix (secret:prod/db vs secret:prod/db-AbCdEf).
• SNS SubscriptionArn format differs from real AWS.
• SQS SendMessage response missing MD5OfMessageBody.
• IAM CreateRole → Role.CreateDate null.

Worked well end-to-end via the CLI (positive)

VPC/subnet/security-group incl. tag-on-create and --filters Name=tag:..., S3 (s3 cp/ls/head-object with real ContentType), DynamoDB (put/get), SQS send/receive round-trip, SNS create/subscribe/publish, IAM role→instance-profile (role attaches), Route 53 zone + change-resource-record-sets, Secrets Manager, SSM Parameter Store (String + SecureString + --with-decryption + get-parameters-by-path --recursive), RDS/Redshift/ElastiCache (endpoints returned), ECS cluster/task-def, EKS (endpoint points to the local k8s data-plane), ELBv2, Bedrock list-foundation-models, CloudWatch Logs write+tail.

Note on protocol-mismatch class (theme)

Finding #1 is likely a class of bug: any service cloudemu implements with a single protocol may break for clients that use a different one. CloudWatch (CBOR-only handler vs query-protocol CLI) is the confirmed instance; worth an audit of each handler's Matches() against what the CLI actually sends.

[thzgajendra]

Update 2: data-plane, deep response validation & remaining-service CLI coverage

Extended the live-server audit to the areas previously flagged as not covered. All against local cloudemu serve only.

Runtime data-planes

• 🔴 Lambda invoke does not execute uploaded code — even with a valid zip + payload it returns StatusCode 200, FunctionError: Unhandled, body {"errorMessage":"no handler registered","errorType":"HandlerError"}. Users can't test function logic.
• ✅ Bedrock Runtime InvokeModel works (simulated) — returns a correctly-shaped Anthropic message ({"type":"message","role":"assistant","content":[{"type":"text",...}],"stop_reason":"end_turn"}).
• ✅ Kubernetes data-plane (:4570) works well — via an EKS cluster endpoint: /version (v1.29.0-cloudemu), create namespace, create Deployment (replicas:2), and pods actually materialize (list pods → 2). Real controller-like behavior.
• ◻︎ SageMaker Runtime InvokeEndpoint reachable; returns NotFound for a missing endpoint (correct). Not exercised against a live endpoint.

Deep response validation (bugs)

• 🟠 DynamoDB Query ignores FilterExpression — Query pk=c1 with FilterExpression amt > 60 returns Count 2 (should be 1). Scan with the same filter correctly returns 1. So the bug is specific to Query — users get unfiltered results silently.
• 🟠 S3 ListObjectsV2 ignores MaxKeys / no pagination — with 5 keys and --max-keys 2 it returns all 5, IsTruncated:false, no NextContinuationToken. Tools that page large buckets break. (Prefix filtering works.)
• ✅ Working: DynamoDB conditional writes (attribute_not_exists → ConditionalCheckFailedException), Query --limit + LastEvaluatedKey pagination, Scan FilterExpression, S3 --prefix, EC2 --filters Name=vpc-id.

Remaining-service CLI coverage

• 🟠 AutoScaling CreateLaunchConfiguration → InvalidAction (not implemented; launch templates exist as the modern path).
• 🟠 ResourceExplorer2 CreateIndex → 405 MethodNotAllowed (+ XML error body on a JSON service). list-views/Search/ListIndexes work.
• ✅ Working via CLI: MemoryDB (create/describe, ARN present), Keyspaces (create/list incl. system keyspaces), NetworkFirewall (create-firewall-policy), Bedrock list-foundation-models, EKS/ELBv2/ECS/RDS/Redshift/ElastiCache create+describe.

Coverage honesty

• A generic every-operation reachability sweep via raw HTTP was attempted but is unreliable — each service needs its exact wire protocol/headers, so raw requests produced false "not-routed" results for services the CLI proved working (IAM/RDS/Redshift/ElastiCache all function via CLI). Trustworthy coverage = SDK-harness (all 30 handlers) + AWS-CLI user flows (~26 services) + data-planes + the deep-validation probes above. A complete per-operation pass would require driving every op of every service through the CLI/SDK — not done.

Net new bugs from this round

1. Lambda invoke doesn't run code (High)
2. DynamoDB Query ignores FilterExpression (Med, silent wrong data)
3. S3 ListObjectsV2 ignores MaxKeys/pagination (Med)
4. AutoScaling CreateLaunchConfiguration missing (Med)
5. ResourceExplorer2 CreateIndex 405 + wrong error body (Med) — same 405/wrong-body class as Lambda TagResource

[thzgajendra]

Update 3: cross-service event delivery, protocol matrix, stateful, and deeper per-op (AWS full push)

🔴 Event delivery is not implemented (wiring stored, events never propagate) — High, systemic

The control-plane wiring succeeds but no events are actually delivered:

• SNS → SQS: subscribe queue + publish to topic → receive-message returns 0 messages.
• EventBridge rule → SQS target: put-targets + put-events → target queue receives 0.
• S3 → SQS notification: PutBucketNotificationConfiguration is mis-routed to CreateBucket (BucketAlreadyOwnedByYou) — the ?notification subresource isn't handled (same class as ?tagging); and object-create delivers 0 to the queue.
• Lambda CreateEventSourceMapping → 405 MethodNotAllowed (not implemented) — no SQS/DynamoDB-stream → Lambda triggers.

Impact: pub/sub fan-out, event-driven architectures, and S3-event pipelines silently do nothing. This is arguably the highest-impact class after the CloudWatch-CLI break.

Protocol matrix — CloudWatch is the sole mismatch (confirmed)

Audited every handler's Matches(). CloudWatch matches only the rpc-v2-cbor URL path, so the CLI's query-protocol request falls through to the EC2 handler (→ InvalidAction, see Update 1). Every other handler matches what the CLI sends. Recommend CloudWatch also accept its query protocol (or disambiguate by the SigV4 monitoring scope).

Stateful/async

• RDS/instances/clusters return terminal state (available) immediately rather than creating → available. SDK wait succeeds instantly. Fine for most uses; differs from real async lifecycle (Low).

Deeper per-op coverage (this round)

EC2 gaps (in addition to Update-1's CreateTags/DeleteTags/CreateNetworkInterface/MonitorInstances/DescribeInstanceStatus/DescribeRegions/DescribeInstanceTypes):

• modify-subnet-attribute (e.g. auto-assign public IP) → InvalidAction — Med, common for public subnets
• create-vpc-endpoint / describe-vpc-endpoints → InvalidAction — consumer-side VPC endpoints missing (PrivateLink service side exists) — Med
• describe-account-attributes, get-console-output → InvalidAction — Low

IAM gaps:

• put-role-policy / list-role-policies (inline role policies) — Med
• tag-role (IAM tagging — tagging theme) — Med
• create-service-linked-role (needed for ECS/EKS bootstrap) — Med

Positive / deep-and-solid: RDS — all 14 deeper ops pass (snapshots, param/subnet groups, clusters, modify, reboot, tags, engine-versions, orderable-options). SageMaker — model/endpoint-config/endpoint/notebook/processing-job/list/describe all pass. These services have real depth.

Still remaining for a truly exhaustive AWS pass

• Every implemented operation of every service driven with valid inputs + field-by-field response parity (only RDS/SageMaker/EC2 now have broad depth).
• Full error-shape/HTTP-status matrix per operation.
• More event-delivery paths (DynamoDB Streams, SNS other protocols, CloudWatch alarm → SNS).
• Bedrock streaming (InvokeModelWithResponseStream/ConverseStream), SageMaker InvokeEndpoint against a live endpoint.

[thzgajendra]

Update 4: exhaustive per-operation sweep + streaming/data-plane (AWS coverage now comprehensive)

Drove many more operations per service via the AWS CLI against the live server, plus the remaining data-planes. Below is the consolidated gap inventory.

🔑 Dominant theme — tagging is missing on ~13 services

CreateTags/TagResource/equivalent returns InvalidAction/UnknownOperation on: EC2, SQS, SNS, DynamoDB, Secrets Manager, SSM, ECR, CloudWatch Logs, EKS, ELBv2, Route 53, Redshift, EventBridge. (Tagging does work on ECS, MemoryDB, Keyspaces, ElastiCache, RDS, and the Resource Groups Tagging API.) A shared tagging surface would fix the biggest single class of user friction.

Per-service gaps found in the deep sweep

Service	Missing (common) ops
SQS	GetQueueAttributes, SetQueueAttributes, SendMessageBatch, ChangeMessageVisibility, PurgeQueue, TagQueue/ListQueueTags, ListDeadLetterSourceQueues — only basic create/send/receive/delete/list/get-url exist
SNS	SetTopicAttributes, PublishBatch, TagResource, CreatePlatformApplication
DynamoDB	UpdateTable, DescribeTimeToLive/UpdateTimeToLive, DescribeContinuousBackups, TagResource/ListTagsOfResource, Streams (see below); Query ignores FilterExpression (Update 2)
Secrets Manager	UpdateSecret, RotateSecret, RestoreSecret, GetResourcePolicy, TagResource
SSM	AddTagsToResource, CreateDocument
ECR	GetAuthorizationToken (docker login), Set/GetRepositoryPolicy, BatchGetImage, TagResource
CloudWatch Logs	PutRetentionPolicy, PutMetricFilter, StartQuery (Insights), TagLogGroup
EC2	CreateTags/DeleteTags, CreateNetworkInterface, MonitorInstances, DescribeInstanceStatus, DescribeRegions, DescribeInstanceTypes, ModifySubnetAttribute, Create/DescribeVpcEndpoints, DescribeAccountAttributes, GetConsoleOutput
IAM	PutRolePolicy/ListRolePolicies (inline), TagRole, CreateServiceLinkedRole
ELBv2	ModifyTargetGroup, AddTags
Route 53	CreateHealthCheck, ChangeTagsForResource
Redshift	CreateClusterParameterGroup, CreateClusterSubnetGroup, CreateTags
ElastiCache	ModifyCacheCluster
EKS	TagResource
EventBridge	TagResource
AutoScaling	CreateLaunchConfiguration
ResourceExplorer2	CreateIndex (405)

Deep & solid (broad op coverage passes): RDS, SageMaker, ECS, MemoryDB, Keyspaces, STS, Bedrock, Bedrock Agent, and the read/CRUD core of every service.

Streaming / data-plane (final)

• ✅ Bedrock Converse works (simulated, correct output.message shape).
• ✅ SageMaker InvokeEndpoint against a live endpoint works (returns InvokedProductionVariant + body).
• ✅ Bedrock InvokeModel (Update 2) and the K8s data-plane (Update 2) work.
• 🔴 DynamoDB Streams not supported — CreateTable --stream-specification is ignored (StreamSpecification: null, no LatestStreamArn) ⇒ no DDB-stream consumers / Lambda triggers.
• 🔴 Lambda invoke doesn't run code (Update 2).
• ◻︎ Bedrock InvokeModelWithResponseStream — inconclusive (CLI arg issue), not verified.

Coverage statement

AWS is now audited comprehensively: all 30 service handlers, the common operation set of each service (deep on the large ones), all data-planes, cross-service event delivery, protocol handling (CLI vs SDK), and stateful behavior. Remaining truly-exhaustive items (every one of the ~700 SDK ops incl. niche variants, and a full per-op error-response-shape matrix) have diminishing returns given the themes above already capture the systemic gaps.

Consolidated top priorities

1. CloudWatch via CLI (query-protocol routing) — Update 1
2. Cross-service event delivery (SNS→SQS, EventBridge→target, S3→SQS, ESM, DDB Streams) — Update 3 + above
3. Tagging across ~13 services — above
4. SQS attributes/batch + ECR auth token + Lambda versioning/aliases/invoke — Updates 1–2 + above
5. DynamoDB Query FilterExpression + S3 ListObjectsV2 MaxKeys correctness — Update 2