CSIT analytics plan through July 31, 2026

[muscariello]

CSIT analytics plan through July 31, 2026

Planning window

• Start date: 2026-05-19
• End date: 2026-07-31
• Execution vehicle: this csit repository

Context

This plan defines the work required to publish clear, reproducible
analytics dashboards for SLIM, Directory, and A2A from this
repository. The emphasis is not generic benchmarking for its own sake.
The emphasis is:

• automated data collection from interoperability and benchmark runs
• scenarios that show practical value for the projects being tested
• analytics dashboards that present the results clearly
• public, reviewable planning and issue tracking

This repository already contains the main building blocks needed for that work:

• local SLIM benchmark automation under benchmarks/agntcy-slim/
• Directory integration tests under integrations/agntcy-dir/tests/
• Directory conformance coverage under
  integrations/agntcy-dir/tests/conformance/
• Directory CI jobs and report artifacts under
  .github/workflows/test-integrations.yaml
• Directory Pages publication support under
  .github/workflows/publish-test-reports-pages.yaml
• A2A interoperability coverage under integrations/agntcy-a2a/
• topology-driven SLIM integration tests under
  integrations/agntcy-slim/
• a local multicluster SLIM testbed under kind-slim-multi-host/

Objective

By 2026-07-31, produce a small but credible CSIT evidence package
that answers:

1. Which CSIT scenarios best demonstrate SLIM, Directory, and
   A2A value?
2. Which metrics should be used to compare those scenarios?
3. Can those scenarios be rerun automatically and published as an
   analytics dashboard on GitHub Pages for agntcy/csit, with
   dedicated views for SLIM, Directory, and A2A?

July exit criteria

The plan is successful only if all of the following are true by
2026-07-31:

• At least one reproducible path exists for SLIM benchmarks,
  Directory tests and conformance, and A2A interoperability.
• At least one multi-cluster or distributed-topology scenario is
  documented and runnable.
• The selected scenarios produce machine-readable artifacts plus a
  human-readable analytics dashboard.
• The published site contains separate dashboard entrypoints for
  SLIM, Directory, and A2A.
• The repo contains a short explanation of why each chosen metric matters.
• The results are published through GitHub Pages in the agntcy/csit
  repository, not left only as workflow artifacts.
• The work is broken into public issues or discussion items so others
  can review and contribute.

Scope

In scope

• selecting a narrow set of high-signal SLIM, Directory, and
  A2A scenarios
• automating repeatable benchmark, Directory test and conformance, and
  interoperability runs
• defining and collecting decision-useful metrics
• generating readable dashboards from raw results
• documenting how to rerun the evidence package
• publishing the final outputs via GitHub Pages in agntcy/csit

Out of scope for this window

• exhaustive protocol coverage across every CSIT component
• full production hardening of test environments
• performance tuning of the underlying projects themselves
• large new infrastructure investments beyond what this repo already supports

Workstreams

1. Scenario selection

Choose a small set of scenarios that answer what should be shown on the
published CSIT analytics site and why.

Target scenario set:

• Local SLIM benchmark baseline
  • use benchmarks/agntcy-slim/
  • purpose: establish repeatable latency and throughput baselines
• Directory integration baseline
  • use integrations/agntcy-dir/tests/
  • purpose: reuse the existing CSIT Directory test suite and surface
    its results in the analytics story
• Directory conformance baseline
  • use integrations/agntcy-dir/tests/conformance/
  • purpose: show client and server conformance results in a form that
    can be published on GitHub Pages
• Cross-SDK interoperability baseline
  • use integrations/agntcy-a2a/
  • purpose: show that heterogeneous agents can communicate reliably
• Distributed topology or multi-cluster SLIM scenario
  • use integrations/agntcy-slim/ and kind-slim-multi-host/
  • purpose: show where SLIM adds value beyond a single-node local
    benchmark

2. Metrics and evidence

Define a short metric set that maps to user value.

Primary metrics:

• end-to-end success rate
• conformance pass rate by supported client and server version
• request latency: mean, p50, p95 or p99
• throughput or completed tasks per second where applicable
• topology-specific delivery success across hops or clusters
• setup reproducibility: one documented command path per scenario

Secondary metrics:

• CPU usage where the existing benchmark suite already exposes it
• run-to-run variance and confidence intervals where repeated runs already exist

Metrics to avoid leading with:

• isolated micro-optimizations with no scenario narrative
• low-level counters without a clear user-facing interpretation

3. Automation and reporting

Use the existing task structure as the default public interface.

Expected command paths:

• task benchmarks:slim:benchmark:suite
• task benchmarks:slim:reports:dashboard
• task integrations:directory:test
• task integrations:directory:tests:client-server:test:all
• task integrations:a2a:test
• task integrations:a2a:reports:dashboard
• topology and multicluster tasks under integrations/agntcy-slim/
  and kind-slim-multi-host/

Target reporting outputs:

• benchmark markdown summary
• benchmark technical report
• HTML dashboards for SLIM, Directory, and A2A artifacts
• reused Directory conformance HTML output staged as
  site/directory/index.html
• one top-level analytics view that summarizes the July CSIT
  scenarios
• one short top-level summary document for non-implementers

4. GitHub Pages publication

The authoritative July deliverable should be a static analytics site
published from the agntcy/csit GitHub Pages setup that already
exists in this repository.

Relevant existing implementation:

• .github/workflows/test-integrations.yaml
• .github/workflows/publish-test-reports-pages.yaml
• .github/workflows/test-benchmarks-slim.yaml
• .github/scripts/build-site-landing-page.sh

Target publication model:

• GitHub Actions produces raw report artifacts
• dashboards are rebuilt in CI into the site/ tree
• existing Directory conformance output is staged directly into
  site/directory/index.html
• GitHub Pages publishes the resulting static files
• the landing page links to scenario-specific dashboards and the
  source workflow runs

Target site structure by the end of July:

• site/index.html: landing page for the published analytics site
• site/a2a/index.html: A2A interoperability dashboard
• site/directory/index.html: Directory conformance dashboard
• site/benchmarks/slim/index.html: SLIM benchmark dashboard
• one July CSIT summary page or section that explains the selected
  scenarios, key metrics, and main conclusions

5. Public planning and issue flow

The execution model should be:

• this plan document for the overall time box
• one issue per concrete deliverable or gap
• discussion threads for scenario and metric selection where feedback is needed

Timeline

Phase 1: bootstrap and scope lock

2026-05-19 to 2026-05-31

Goals:

• publish the execution plan
• inventory the runnable benchmark, existing Directory test and
  conformance flows, interoperability, and topology flows already
  present
• reduce the scenario list to the smallest useful set
• define the first metric table with plain-language rationale

Deliverables:

• this plan committed under docs/plans/
• a shortlist of July target scenarios
• an issue backlog grouped by workstream
• a publication target defined for GitHub Pages

Definition of done:

• the team agrees on the scenario set
• each scenario has an owner and an issue
• no major ambiguity remains about the July objective

Phase 2: baseline automation

2026-06-01 to 2026-06-14

Goals:

• verify that the local SLIM benchmark path is reproducible
• verify that the existing Directory integration test path is
  reproducible
• verify that the Directory conformance path is reproducible
• verify that the A2A interoperability path is reproducible
• confirm the dashboard rebuild and Pages publication flow for all
  three dashboard areas

Deliverables:

• first successful benchmark run with saved artifacts
• first successful Directory integration test artifact
• first successful Directory conformance run with saved artifacts
• first successful interoperability run with saved artifacts
• first successful GitHub Pages staging path using existing workflows
• a documented rerun recipe for all three dashboard areas

Definition of done:

• a contributor can execute the baseline paths from documented commands
• output artifacts are stored in predictable locations
• the CI path from artifact generation to site/ output is understood
• obvious gaps in task entrypoints or docs are captured as issues

Phase 3: SLIM distributed-topology evidence

2026-06-15 to 2026-06-30

Goals:

• choose one distributed or multi-cluster topology that best
  illustrates SLIM value
• make the scenario runnable with the current CSIT environment
• capture the first comparative evidence set

Candidate focus:

• multi-cluster control or routing behavior using kind-slim-multi-host/
• topology-driven routing validation using integrations/agntcy-slim/

Deliverables:

• one runnable topology scenario
• one short note describing why this scenario matters
• first-pass data for success rate, latency, and observed routing behavior

Definition of done:

• the team can rerun the scenario without manual tribal knowledge
• the scenario clearly demonstrates more than a single-node benchmark

Phase 4: tighten metrics and comparisons

2026-07-01 to 2026-07-17

Goals:

• refine the chosen metrics so they support a clear argument
• rerun scenarios enough times to produce stable summary data
• improve dashboards or summaries where the raw output is still too
  hard to consume
• shape the GitHub Pages presentation around the selected July story

Deliverables:

• updated metric definitions
• repeated runs for the selected scenarios
• draft narrative connecting the metrics to SLIM, Directory, and
  A2A value
• draft GitHub Pages dashboard layout or content plan

Definition of done:

• the evidence package is readable by someone who did not run the tests
• repeated runs are sufficient to avoid anecdotal conclusions

Phase 5: publishable July package

2026-07-18 to 2026-07-31

Goals:

• finalize the July evidence package
• publish a concise analytics dashboard for internal and external review
• turn remaining gaps into August follow-up items instead of leaving
  them implicit

Deliverables:

• final July GitHub Pages dashboard
• final July summary document
• published dashboard entrypoints for SLIM, Directory, and A2A
• linked dashboards and raw artifacts
• issue list for the next phase

Definition of done:

• the repo contains a coherent story, not just raw artifacts
• reviewers can understand what was measured, why it matters, and how
  to rerun it
• the published GitHub Pages site is the primary entrypoint for July results
• the team has an explicit decision on what moves into August

Suggested issue breakdown

• plan: maintain the execution plan and keep dates current
• benchmarks: stabilize the SLIM benchmark run path and report outputs
• directory: stabilize the existing Directory integration and
  conformance run paths and published outputs
• interop: stabilize the A2A run path and report outputs
• topology: select and automate the July SLIM topology scenario
• metrics: define metric semantics and acceptance criteria
• dashboard: shape the Pages site, landing page, and cross-project
  summary view
• docs: write the July summary and rerun instructions

Dependencies and risks

Dependencies

• access to the required local tooling for task execution
• stable task entrypoints for benchmark, existing Directory test and
  conformance jobs, and interoperability suites
• enough environment reliability to rerun scenarios without
  hand-fixing the setup each time

Risks

• trying to cover too many scenarios will dilute the July outcome
• multi-cluster setup complexity can consume the schedule if used
  before the baseline paths are stable
• dashboards may exist but still fail to answer the actual "why this
  project?" question for SLIM, Directory, or A2A
• Pages publication may succeed technically while still presenting
  fragmented results

Risk response

• keep the July scenario set intentionally small
• finish baseline automation before expanding topology complexity
• require every metric to have a one-line value statement
• treat the GitHub Pages dashboard as a product, not as an
  afterthought on top of raw reports