feat: add JSONL draft diff logic and update command support by nborges-aws · Pull Request #1926 · aws/agentcore-cli

nborges-aws · 2026-08-05T23:27:45Z

Description

Adds eval dataset update, which reconciles a local JSONL draft file into the remote dataset DRAFT. Update is append-only at the service level, so the CLI cannot replace the JSONL file wholesale. Instead, we have to compute a diff between the local draft and the remote draft, then apply the required dataset mutations:

add examples missing from the remote draft
update examples where exampleId exists remotely but has content changes
delete remote examples no longer present in local draft
ensure untouched examples remain intact

Summary of changes:

Adds eval dataset update --id <dataset-id> --file-path <path>.
Downloads the remote DRAFT through the presigned downloadUrl from GetDataset. I considered using the ListDatasetExamples API for this, but this would've required N API calls, where N is ceil(exampleCount / pageSize).
Adds dataset diff logic for classifying additions, updates, deletes, and unchanged rows.
Writes exampleId for newly added examples back into the local JSONL file, so future updates reconcile correctly.
Validates local JSONL to ensure valid JSON objects, exampleId as non-empty strings, and rejects duplicate exampleId
Validate remote exampleId's are unique and responses have one ID per example
Adds shared IO helpers for JSONL parsing and generic text file reads
Move dataset download plumbing into fetch method, to be reused between eval dataset get --file-path and update’s remote draft fetch
Adds polling logic to await dataset to return ACTIVE during update process. Since update is a series of batch mutations, this is necessary. Otherwise, updates would fail if status was "UPDATING/DELETING" etc. during an attempted update

Related Issue

Closes #

Documentation PR

Type of Change

Testing

Manually tested commands using bun run ... against personal account with following procedure:

Created dataset from local JSONL (and waited until ACTIVE)
downloaded remote draft
edited local JSONL with one add, one update, one delete
ran eval dataset udpate
confirmed returned counts: added: 1, updated: 1, deleted: 1, unchanged: 0
confirmed local JSONL rewritten with new exampleId
downloaded remote draft, confirmed it matched the updated local file
published the updated draft

Full suite against latest HEAD after rebasing onto refactor:

bun test (851 pass, 0 fail)
bun run typecheck clean
bun run lint:check clean
bun run format:check clean
bun run build clean

How have you tested the change?

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

codecov-commenter · 2026-08-05T23:29:15Z

Codecov Report

❌ Patch coverage is 99.80237% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.95%. Comparing base (9d63f5b) to head (934b59a).
⚠️ Report is 1 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/datasetDiff.ts	99.09%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1926      +/-   ##
============================================
+ Coverage     96.88%   96.95%   +0.06%     
============================================
  Files           342      346       +4     
  Lines         19263    19737     +474     
============================================
+ Hits          18663    19136     +473     
- Misses          600      601       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jariy17

LGTM, but I would add onProgress bar

jariy17 · 2026-08-06T21:33:32Z

+          `expected a non-empty string`,
+        { meta: { line: lineNumber } },


I don't know if lineNumber would be useful metadata for telemetry.

jariy17 · 2026-08-06T22:04:13Z

+      flag("id", "the ID of the dataset to update", z.string().optional()),
+      flag("file-path", "local JSONL file to reconcile into the DRAFT", z.string().optional()),
+    ],
+    handle: async (ctx, flags) => {


runDatasetExampleBatches runs batches serially and polls for ACTIVE after each one (up to 60s per batch). A large diff sits with no output for minutes, so the command looks hung. We already have the onProgress pattern in project/manager.tsx. Can we wire the batch loop into it and print "Applying update"?

Great call out! I hooked up the onProgress pattern, and it improves the experience alot while waiting on updates to complete. PR has been updated with changes

jariy17

We need golden tests for update flow.

jariy17 · 2026-08-07T17:06:30Z

+  return `${rows.map((r) => JSON.stringify(r)).join("\n")}\n`;
+}
+
+describe("parseJsonl", () => {


Do we need this tests? Will the handler tests cover this for us?

The handler test covers the CLI routing and flag handling. It doesn't directly exercise the diff calculation and rules, which is what these tests are for.

jariy17 · 2026-08-07T17:07:07Z

+  return { control: () => client, data: () => client, iam: () => client };
+}
+
+describe("EvalClient.updateDatasetExamples", () => {


Do we need these unit tests? Won't the handler tests cover this?

Same as above for handler. These tests cover the delete/update/add mutations, batching, polling logic, etc. Different surface than handlers

jariy17

Nice job. The update diff is pretty good!

aidandaly24 · 2026-08-10T16:16:01Z

+  },
+  "description": "Recorded fixture for the dataset commands",
+  "draftStatus": "MODIFIED",
+  "downloadUrl": "https://agentcoredatasets685197708687-284077270265-us-west-2-an.s3.us-west-2.amazonaws.com/685197708687/datasets/agentcore_cli_dataset_fixture-jzVpQaA5It/draft/dataset.jsonl?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJIMEYCIQCflkdW2nGM6BMjGlmG2AV9i5D4WezB625lTpjdcrGZ6wIhAL6vK33WybFtnZ0UP%2BgXuKospslfqDD69y7%2Fzv0XXn%2BlKvIECFwQABoMMjg0MDc3MjcwMjY1IgzOAcYPrFp9BqEy2NQqzwQHK90X3idFVQP4puJ2qijfTwzzcj9xbawSUEAFYlrfw45YamqnalNNRxfK%2BeQmgA8Tj5QWWNn3noXXcQ8%2BFSwmqEkLElukt5IVBH59sUvJ13vjVUH%2Fsah%2FpCl9%2FzNs7O7rDnn0QrKQvGEHButw2ftwWobC9cijfwZWP1KXQc1hj8gLxpeNjQSH8RWAnGYsHL%2FwQVkg9AUr4Fc1sPFrdmH22Kyla62M5sJ0%2FBgB%2FsSSSqeG%2B%2FE7egRt5zCHDScu0%2FydT0GRzEkUc5TUZiB1wRi1lHRudPGfLuUGAr27Gr5RfhUvgwceu4AY6ShiBOhh8djgKD%2B7uqtgj41P4Hxg%2Ft%2FZY5PTYH9xATKM4CTyNZ6HO57xHeo2M%2BmeGIF3nLe7I53ruMq4onogm8srajiAhs1%2FPHxSSXx5ga4MwtB9pE%2BsM26JWD3QIVdu6T%2FAK5Y0CBvji2PP1jnr89nS%2B10FTYULZJn1DD7P%2F7R2idxvKDFzKY6z7tluDo3yybXSICtjapW0A72cg0vAzXrl6DKlZuZF6S9WEyKUcmDJeznwRqbSgrezL%2Bu2utpIcMdhfGsVaEvFxLuU5M9YkC7KA6rXJtn2zdm80olJN3EgOwIF%2Bf3FwBUxPCsd1F0JM4RTQC%2F4cuObwJ7hUHHZkodbAXjKPu%2B2hEGBtZZOJOB78O7liJ7xlRSaeMQl3ZETMcLKerIdlzKRRr%2BV%2F29yKHRx8TZI08Y4KoXaqrHmJi9ZaXwa8wwBwckMcrQ%2B6%2FWrWtitd6WCvzJeMwZ%2FaZEso59nRnxYtQww9sbY0wY6ogEFvsbxfePrmzBMVs4DXNp%2F3WCIpofAAPDT0lPsWqFcN8deD1qpDK4gOwprTsurb6NLaI1fjVnAFXEeYesR8WaPOvFHEIDVGh3U4uCWU9nDd4VRJFOaXGkunU5JMCfOBVCQpWU5bazXNdS3aJcr0MAhy62Ii%2BHJC5nm20JTPU%2BcEwKMxnfQVfl1b9Yr80zEWp52Y5G6DfFjb3XYuUu8HgDQCic%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260807T190246Z&X-Amz-SignedHeaders=host&X-Amz-Credential=ASIAUEJCTET4RAA4PZHK%2F20260807%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Expires=300&X-Amz-Signature=ac646581e361f5b80fce8df15655aae1cc74921c8a7658272522788555d58634",


Could we sanitize presigned URLs before persisting fixtures? The changed GetDataset and golden fixtures contain the full X-Amz-Security-Token, credential scope, and signature. They are short-lived, but we still should not commit them publicly. fixtureFetch already keys by the stable pathname, so would it make sense to store a redacted copy while returning the real URL during recording, and redact the corresponding golden output?

aidandaly24 · 2026-08-10T16:17:12Z

+        try {
+          // The remote request has already succeeded, so checkpoint its IDs even
+          // if cancellation arrives before the next poll or batch.
+          await atomicWrite(filePath, nextLocalText);


I was wondering if we need an optimistic check before replacing the local file here. The command may run for several minutes, but each checkpoint is rebuilt from the initial localExamples snapshot. I reproduced editing the JSONL while the Add request was running, and this write silently replaced that edit with the original row plus its assigned ID. Could we verify that the file still matches the last known contents before replacing it, and preserve the reconciled output separately if it changed?

Yeah adding this check makes total sense. I've updated the logic so we verify the expected state of the local file. If the file has changed during update we leave it untouched. Instead, we write the reconciled output to a separate recovery file, surface that files' path to the user, and stop before the next batch.

aidandaly24 · 2026-08-10T16:18:03Z

+    // Build every batch before mutating the remote draft so an oversized
+    // individual example cannot fail after earlier phases have already run.
+    const deleteBatches = buildDatasetExampleBatches({
+      items: diff.deleteIds,
+      payloadItem: (exampleId) => exampleId,
+      requestBody: (exampleIds, clientToken) => ({ exampleIds, clientToken }),
+    });


Should the payload-size calculation include datasetId? These requestBody callbacks size only the examples and client token, while the actual SDK commands also include datasetId. I constructed a batch accepted as exactly 5 MB here whose actual command input was 115 bytes over the limit. Would it make sense to size the complete command input for all three mutation types so the preflight guarantee is accurate?

Great catch; absolutely should factor datasetId into the calc. Updated

aidandaly24 · 2026-08-10T16:18:51Z

    const path = fixturePath(dir, command);

    if (isRecording()) {
+      const shouldWrite = !recordedPaths.has(path);


Could this first-response behavior break existing record/replay flows that poll the same request? For example, the Harness fixture records repeated GetHarness calls and relies on the final READY response being left in the fixture. With this global first-write rule, a fresh RECORD=1 run can preserve the initial CREATING response, and the next offline replay fails because the fixture is not settled. Would it make sense to support response sequences or scope this behavior to the Dataset update fixture?

Good callout. I reset the last-response behavior in the shared recorder. Also moved the update fixture into its own directory, which prevents overwriting fixtures from the get fixture. This fixes the initial issue which made me switch to first reponse behavior originally

jariy17

LGTM

…state when writing

github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 5, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 5, 2026

nborges-aws force-pushed the datasets-update-cli branch from e027664 to 45eb79e Compare August 6, 2026 20:38

Base automatically changed from datasets-cli to refactor August 6, 2026 20:39

nborges-aws force-pushed the datasets-update-cli branch from 45eb79e to e23acd1 Compare August 6, 2026 20:39

jariy17 requested changes Aug 6, 2026

View reviewed changes

nborges-aws force-pushed the datasets-update-cli branch 2 times, most recently from 9340e02 to 1214e97 Compare August 7, 2026 15:32

jariy17 reviewed Aug 7, 2026

View reviewed changes

nborges-aws force-pushed the datasets-update-cli branch from 1214e97 to 671780e Compare August 7, 2026 19:04

jariy17 previously approved these changes Aug 7, 2026

View reviewed changes

aidandaly24 reviewed Aug 10, 2026

View reviewed changes

nborges-aws dismissed jariy17’s stale review via 41bd1a1 August 13, 2026 14:34

nborges-aws force-pushed the datasets-update-cli branch from 671780e to 41bd1a1 Compare August 13, 2026 14:34

jariy17 previously approved these changes Aug 13, 2026

View reviewed changes

aidandaly24 previously approved these changes Aug 13, 2026

View reviewed changes

nborges-aws force-pushed the datasets-update-cli branch from d7ff900 to ebdc34e Compare August 13, 2026 16:23

nborges-aws dismissed stale reviews from aidandaly24 and jariy17 via 3e08721 August 13, 2026 16:27

nborges-aws force-pushed the datasets-update-cli branch 2 times, most recently from 3e08721 to 4203c52 Compare August 13, 2026 16:27

jariy17 approved these changes Aug 13, 2026

View reviewed changes

aidandaly24 approved these changes Aug 13, 2026

View reviewed changes

nborges-aws added 5 commits August 13, 2026 12:33

feat: add JSONL draft diff logic and update command support

a623eea

fix: precompute all batches and add incremental ID persistence

5cbe2a4

fix: model MalformedResponseError

14b8880

chore: add onProgress pattern to dataset update batches

887e091

chore: add fixture test for dataset update flow

bf02611

nborges-aws added 2 commits August 13, 2026 12:33

fix: remove first-response behavior, sanitize URLs, and verify draft …

e10cb7b

…state when writing

fix: add missing logs stub in update test

934b59a

nborges-aws force-pushed the datasets-update-cli branch from 4203c52 to 934b59a Compare August 13, 2026 16:33

nborges-aws merged commit ca6426d into refactor Aug 13, 2026
8 checks passed

nborges-aws deleted the datasets-update-cli branch August 13, 2026 16:34

		`expected a non-empty string`,
		{ meta: { line: lineNumber } },

Uh oh!

Conversation

nborges-aws commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Documentation PR

Type of Change

Testing

Checklist

Uh oh!

codecov-commenter commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jariy17 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nborges-aws commented Aug 5, 2026 •

edited

Loading

codecov-commenter commented Aug 5, 2026 •

edited

Loading