feat: launch telemetry by Hweinstock · Pull Request #1430 · aws/agentcore-cli

Hweinstock · 2026-06-01T14:03:12Z

Description

This is a composite PR of a few changes.

add config command, see feat: add config command with dot-notation support #1387 for more info.
fix some misc bugs related to telemetry. Ex. .json file extension is misleading for .jsonl files, invalid configs silently overwritten. (ex. [bug]: invalid agentcore config is silently overwritten. #1337).
publish telemetry endpoint to config.
add documentation in the help command for how to manage telemetry settings and use audit mode.
fix a bug where telemetry notice showed every time when config is malformed.
address some testing gaps in telemetry/config.

Future Work (non-launch blocking):

add a page to docs/ for how to use telemetry audit mode.

Related Issue

Closes #

Documentation PR

Will follow-up with telemetry documentation.

Type of Change

Testing

How have you tested the change?

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

…verrides

…entcore config

…valid UUID

github-actions · 2026-06-01T14:04:12Z

Package Tarball

aws-agentcore-0.16.0.tgz

How to install

gh release download pr-1430-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.16.0.tgz

agentcore-devx-automation · 2026-06-01T14:06:10Z

Claude Security Review: no high-confidence findings. (run)

agentcore-cli-automation

Overall the launch-telemetry changes look solid — moving telemetry preferences under a generic config command, adding a Result-based read API, and validating endpoints/UUIDs are good cleanups. I flagged a few items below that I think need to be addressed before merging — primarily missing telemetry instrumentation on the new config command itself (which is required per src/cli/telemetry/README.md), and stale entries in COMMAND_SCHEMAS for the removed telemetry enable/disable subcommands.

agentcore-cli-automation · 2026-06-01T14:17:42Z

+      const result = await resolveAction(key, value)();
+      printResult(result);
+      if (!result.success) process.exit(1);
+    });


Missing telemetry instrumentation for the new config command.

This PR is titled "launch telemetry" and the README at src/cli/telemetry/README.md requires every new top-level command to (a) declare an attribute schema in src/cli/telemetry/schemas/command-run.ts under COMMAND_SCHEMAS, and (b) be wrapped with runCliCommand (or withCommandRunTelemetry) so a cli.command_run metric is emitted. The action handler here just runs resolveAction(...), prints, and exits — no metric is ever recorded for agentcore config ... invocations, which is precisely the kind of usage we'd want visibility into now that config is the documented way to opt in/out of telemetry.

A few options:

Add three command keys (config.list, config.get, config.set) to COMMAND_SCHEMAS (most likely all NoAttrs, or config.set could carry a sanitized key enum if we want to track which keys users touch most), and wrap each branch with runCliCommand.

Add a single config key to COMMAND_SCHEMAS and dispatch all three branches through one runCliCommand('config', ...) call, with attrs distinguishing the action.

Option 1 is more consistent with how dataset.*, import.*, etc. are modeled today.

will follow-up with this, I don't think its a blocker and I'd like to sync the backend schemas once I have a shape.

agentcore-cli-automation · 2026-06-01T14:17:42Z

-export async function readGlobalConfig(configFile = GLOBAL_CONFIG_FILE): Promise<GlobalConfig> {
+export function validateGlobalConfig(data: unknown): { success: boolean; error?: z.ZodError } {
+  return GlobalConfigSchemaStrict.safeParse(data);
+}


validateGlobalConfig returns { success: boolean; error?: z.ZodError }, which loses the discriminated union that safeParse provides. Callers like handleConfigSet only branch on success, so this works, but in the success case any caller that wants the parsed/coerced value gets nothing — and TS won't tell you that the data field is gone. Either:

Return z.SafeParseReturnType<unknown, GlobalConfig> (i.e., just GlobalConfigSchemaStrict.safeParse(data) typed as-is), or

Return an explicit Result<{ data: GlobalConfig }, ZodError> for consistency with the rest of the file.

This matters because today handleConfigSet writes partial (the user-supplied object) rather than validation.data — so if we ever add a .transform() or coercion to the schema, the validated/normalized form would be silently discarded.

this is intentional. validate is used for validation, read is used for reading. Validate is also used to verify that new (unwritten) config shapes are valid. The purpose of the function is "is this shape a valid config", so a success boolean is sufficient.

agentcore-devx-automation · 2026-06-01T14:43:04Z

Claude Security Review: no high-confidence findings. (run)

Hweinstock · 2026-06-01T14:13:01Z

 } from '../../lib/schemas/io/global-config';
 import { createTempConfig } from './helpers/temp-config';
 import { readFile, writeFile } from 'fs/promises';
+import assert from 'node:assert';


note: we import and use assert here because it provides type narrowing and avoids verbosity in follow-up assertions on result types.

Hweinstock · 2026-06-01T14:14:01Z

+export async function handleConfigGet(key: string): Promise<ConfigResult> {
+  const read = await readGlobalConfig();
+  if (!read.success) {
+    return { success: false, error: new Error(`Error: Unable to parse config file at ${GLOBAL_CONFIG_FILE}`) };


note: planning to follow-up with some telemetry on the config command and make this a custom error. Could use ValidationError, but may want something more specific to the file system.

Hweinstock · 2026-06-01T14:15:24Z

+
+  For more information on what exactly is captured, see the schemas, which
+  include all attributes and metrics captured:
+    https://github.com/aws/agentcore-cli/tree/main/src/cli/telemetry/schemas


once we have proper documentation, I'll replace the source code link with a docs link.

Hweinstock · 2026-06-01T14:17:01Z

+    try {
+      sinks.push(new OtelMetricSink({ endpoint, resource }));
+    } catch {
+      // Telemetry is best-effort — skip the network sink rather than crash.


this is needed for the case where a customer provides an invalid url in their config or via env var.

Hweinstock · 2026-06-01T14:17:38Z

  validate: 'Validate agentcore/ config files.',
  'config-bundle': '[preview] Manage configuration bundle versions and diffs.',
  archive: '[preview] Archive (delete) a batch evaluation or recommendation on the service and clear local history.',
+  config: 'Adjust global configuration settings such as telemetry opt-out status',


note: should move these somewhere more intuitive, but OOS here.

Hweinstock · 2026-06-01T14:21:12Z

+      const result = await resolveAction(key, value)();
+      printResult(result);
+      if (!result.success) process.exit(1);
+    });


will follow-up with this, I don't think its a blocker and I'd like to sync the backend schemas once I have a shape.

Hweinstock · 2026-06-01T14:26:18Z

-export async function readGlobalConfig(configFile = GLOBAL_CONFIG_FILE): Promise<GlobalConfig> {
+export function validateGlobalConfig(data: unknown): { success: boolean; error?: z.ZodError } {
+  return GlobalConfigSchemaStrict.safeParse(data);
+}


this is intentional. validate is used for validation, read is used for reading. Validate is also used to verify that new (unwritten) config shapes are valid. The purpose of the function is "is this shape a valid config", so a success boolean is sufficient.

avi-alpert · 2026-06-01T15:33:42Z

+describe('config command', () => {
+  afterAll(() => rm(testConfigDir, { recursive: true, force: true }));
+
+  it('lists config with only installationId when fresh', async () => {


the "only" is misleading - this test checks installationId is there but not that the config only has that key

good catch, updated.

avi-alpert · 2026-06-01T15:45:27Z

@@ -214,8 +214,6 @@ export const COMMAND_SCHEMAS = {
  'dataset.publish-version': NoAttrs,
  'dataset.remove-version': NoAttrs,
  'telemetry.disable': NoAttrs,


was this removed?

agentcore-devx-automation · 2026-06-01T16:35:39Z

Claude Security Review: no high-confidence findings. (run)

Hweinstock added 18 commits June 1, 2026 12:56

feat: instrument config command

9aa6670

fix: simplify set path by validing the partial instead of the merged

a977e51

refactor: unify parsing logic with telemetry for common abstraction

24de6c4

fix: adjust tests to include quotes for strings

76e6890

docs: update display text that telemetry is now active

0f33f91

fix(config): avoid overwriting config when not parsable

a23b8b5

feat(telemetry): wire endpoint resolution with constant default and o…

7b861f3

…verrides

test(config): exit non-zero on corrupt config

ef5daa1

fix(config): show friendly error with path on config read/write failures

b919c75

refactor(telemetry): remove enable/disable subcommands in favor of ag…

2df5774

…entcore config

fix(config): regenerate installationId when persisted value is not a …

e112e68

…valid UUID

fix(telemetry): suppress first-run notice when telemetry is disabled

fd9606e

test(integ): isolate spawned CLI from host telemetry env

073fedc

fix(telemetry): use .jsonl extension for audit files

a2f92f5

docs(telemetry): correct stale comment on OtelMetricSink try/catch

b11dff7

test(config): remove redundant integ tests and fix type error

01ea06c

docs: strip noisy comments from telemetry instrumentation

16042d4

fix(test): update telemetry helper to search the correct files

75b9a58

github-actions Bot added the size/l PR size: L label Jun 1, 2026

Hweinstock temporarily deployed to e2e-testing June 1, 2026 14:03 — with GitHub Actions Inactive

github-actions Bot added the agentcore-harness-reviewing AgentCore Harness review in progress label Jun 1, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 1, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Jun 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 1, 2026

Hweinstock mentioned this pull request Jun 1, 2026

feat: add config command with dot-notation support #1387

Closed

11 tasks

agentcore-cli-automation suggested changes Jun 1, 2026

View reviewed changes

Hweinstock added 2 commits June 1, 2026 14:22

chore(telemetry): remove dead commands from schema

f60789d

docs: remove double quote in comment

95ccc8f

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Jun 1, 2026

Hweinstock temporarily deployed to e2e-testing June 1, 2026 14:27 — with GitHub Actions Inactive

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 1, 2026

Hweinstock commented Jun 1, 2026

View reviewed changes

Hweinstock marked this pull request as ready for review June 1, 2026 14:46

Hweinstock requested a review from a team June 1, 2026 14:46

avi-alpert reviewed Jun 1, 2026

View reviewed changes

avi-alpert previously approved these changes Jun 1, 2026

View reviewed changes

fix: swap telemetry tests to existing metric

a59d425

Hweinstock dismissed avi-alpert’s stale review via a59d425 June 1, 2026 16:29

github-actions Bot added size/xl PR size: XL and removed size/l PR size: L labels Jun 1, 2026

Hweinstock temporarily deployed to e2e-testing June 1, 2026 16:29 — with GitHub Actions Inactive

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 1, 2026

Hweinstock temporarily deployed to e2e-testing June 1, 2026 16:42 — with GitHub Actions Inactive

avi-alpert approved these changes Jun 1, 2026

View reviewed changes

Hweinstock merged commit 0e86605 into aws:main Jun 1, 2026
34 of 48 checks passed

Conversation

Hweinstock commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Documentation PR

Type of Change

Testing

Checklist

Uh oh!

github-actions Bot commented Jun 1, 2026

Package Tarball

How to install

Uh oh!

agentcore-devx-automation Bot commented Jun 1, 2026

Uh oh!

agentcore-cli-automation left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agentcore-devx-automation Bot commented Jun 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agentcore-devx-automation Bot commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Hweinstock commented Jun 1, 2026 •

edited

Loading