Skip to content

fix: order AgentCore Memory events at millisecond resolution#572

Merged
jariy17 merged 2 commits into
mainfrom
fix/monotonic-timestamp-multiprocess
Jul 9, 2026
Merged

fix: order AgentCore Memory events at millisecond resolution#572
jariy17 merged 2 commits into
mainfrom
fix/monotonic-timestamp-multiprocess

Conversation

@jariy17

@jariy17 jariy17 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #571.

Problem

The monotonic timestamp counter added in #83 increments by 1 second on collision and stores its state at class level. Two consequences:

  1. Events get pushed seconds into the future. Every same-second collision bumps the timestamp by a full second, so a turn that writes several events in quick succession stamps them 1s, 2s, 3s… ahead of wall-clock. The premise behind the 1-second step ("boto3 doesn't support sub-second resolution") does not hold: the bedrock-agentcore client serializes eventTimestamp as a unix timestamp with millisecond precision, and the service preserves it (the eventId prefix is a millisecond epoch).

  2. State is shared across unrelated sessions. Because _last_timestamp and the lock are class attributes, concurrent session managers for different sessions in the same process perturb each other's ordering.

This future-dating is what makes events reorder when a session is written by more than one process (e.g. consecutive turns served by different pods): a turn's events end up several seconds ahead, and a later turn written from a fresh process at real wall-clock time can sort before them, interleaving the two turns. Interleaving can place a toolResult before its toolUse, which the Converse API rejects.

Changes

  • Break ties at 1 millisecond instead of 1 second — the resolution AgentCore Memory actually stores and orders eventTimestamp at. A colliding timestamp advances by 1ms rather than inflating by a full second, so a multi-event turn stays within milliseconds of real time instead of drifting seconds ahead.
  • Make the timestamp state instance-scoped (per-manager lock + _last_timestamp) so concurrent managers for different sessions no longer share or perturb each other's ordering.

Behavior is otherwise unchanged: the first timestamp and any timestamp later than the current floor still pass through untouched; None still falls back to datetime.now(timezone.utc).

Scope

This keeps ordering correct within a process and removes the multi-second future-drift that caused cross-process interleaving in practice. It does not add explicit cross-process coordination — a freshly started process still begins from wall-clock time — but because events are no longer pushed seconds into the future, sequential turns handed off between processes with any real gap no longer collide. Guaranteeing ordering under genuinely concurrent writers to the same session would require server-side sequencing and is out of scope here.

Testing

Unit testsTestMonotonicTimestamp (6 tests): instance-scoping (state not shared across managers), 1ms-not-1s tie-break, passthrough of the first / a later / a None timestamp, and a same-second burst that stays strictly ordered at millisecond spacing. Full memory-integration module: 178 passed locally (2 unrelated async tests require a pytest-asyncio plugin not installed locally; unaffected by this change and green in CI).

Multi-process experiment against real AgentCore Memory (subprocess-per-"pod"): with the old +1s counter, a 5-event turn drifted ~4s into the future and a follow-up turn from a fresh process interleaved deterministically (0/10 ordered). With the 1ms tie-break the same scenario stays ordered (10/10). A real Strands agent + real Bedrock model run confirmed the counter drifts a turn ~1.5–3.7s into the future on main (each collision = +1s), which this change eliminates.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

✅ No Breaking Changes Detected

No public API breaking changes found in this PR.

@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jul 9, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jul 9, 2026
jariy17 added 2 commits July 9, 2026 17:24
The monotonic timestamp counter incremented by 1 second on collision and
stored state at class level. The 1s increment pushed a busy turn's events
several seconds into the future (boto3/AgentCore Memory actually resolve
eventTimestamp at millisecond granularity, not 1s), and class-level state
was shared across managers for different sessions in the same process.

Break ties at 1ms instead of 1s, and make the timestamp state instance-scoped.
@jariy17 jariy17 force-pushed the fix/monotonic-timestamp-multiprocess branch from da96548 to b94474a Compare July 9, 2026 17:24
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jul 9, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jul 9, 2026
@jariy17 jariy17 changed the title fix: order AgentCore Memory events at millisecond resolution and coordinate across processes fix: order AgentCore Memory events at millisecond resolution Jul 9, 2026
@jariy17 jariy17 merged commit a271ab4 into main Jul 9, 2026
41 checks passed
@@ -115,20 +113,12 @@ def _get_monotonic_timestamp(cls, desired_timestamp: Optional[datetime] = None)
if desired_timestamp is None:
desired_timestamp = datetime.now(timezone.utc)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should floor the desired_timestamp, so two events within the same millisecond count as a tie. Timestamp as set here will be in microseconds, and comparison on line 117 will evaluate as a tie if within the same millisecond, but not the same microsecond.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, you're right. datetime.now() gives microseconds but the service only keeps milliseconds, so two events in the same ms but different microseconds would pass the tie check and then collide once stored.

Fixed it by flooring desired_timestamp to ms before the comparison. Now same-ms events are treated as a tie and bumped 1ms apart. Added a regression test for the same-ms/different-microsecond case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AgentCoreMemorySessionManager reorders events across processes because of the 1-second monotonic counter

3 participants