feat(imports): cache remote stack-import clones (dedup + opt-in TTL)#2571
Merged
Merged
Conversation
Clone each remote (git) stack-import source repo at most once per Atmos invocation instead of once per import, and add an opt-in `ttl` to reuse the clone across runs (so a warm CI cache can skip the re-clone). - Wire the default git-subdir resolve path through ensureSourceDir so all subdir imports of the same repo share a single clone (within-run dedup, spanning both describe-affected passes via the global importer). - Add per-session fetch tracking + TTL-based cross-run freshness to ensureSourceDir; persist UpdatedAt in the .atmos-source-ready marker. No ttl = refresh once per run, so mutable refs (?ref=main) stay fresh. - Expose `ttl` per-import (StackImport.TTL) and as a global imports.ttl default; thread it through ResolveRemoteImportNested. - Extract shared duration.IsExpired/IsZeroTTL; source provisioner now delegates to them. - Update JSON schemas, add tests, docs, changelog post, and roadmap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Erik Osterman (Cloud Posse) (osterman)
requested a review
from a team
as a code owner
Contributor
|
Tip Atmos Pro
No affected stacks workflow was detected for this pull request. 
|
Erik Osterman (Cloud Posse) (osterman)
added
the
minor
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
Contributor
|
Lost in the diff? Review this PR in Change Stack to follow the change map from intent to exact ranges. Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughRemote stack imports now support configurable caching via a new ChangesRemote Stack Import Caching with TTL
🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
website/src/data/roadmap.js (1)
323-323: ⚡ Quick winMilestone structure looks solid.
The new "Remote import source caching" entry includes all required fields—status, quarter, changelog, docs—and the description accurately reflects the within-run dedup and opt-in TTL behavior. Consider adding
pr: 2571to link directly to this pull request for traceability.➕ Optional: add PR link
- { label: 'Remote import source caching (clone once + TTL)', status: 'shipped', quarter: 'q2-2026', changelog: 'faster-remote-stack-imports', docs: '/stacks/imports#caching-remote-imports', description: '...' }, + { label: 'Remote import source caching (clone once + TTL)', status: 'shipped', quarter: 'q2-2026', pr: 2571, changelog: 'faster-remote-stack-imports', docs: '/stacks/imports#caching-remote-imports', description: '...' },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@website/src/data/roadmap.js` at line 323, Add a pr field linking the PR number to the roadmap entry by updating the object with label 'Remote import source caching (clone once + TTL)' to include pr: 2571 (e.g., add pr: 2571 alongside status, quarter, changelog, docs, description, benefits) so the roadmap row references the originating pull request for traceability.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/duration/duration.go`:
- Around line 182-184: The zero-value TTL matching in the switch on ttl fails
for inputs with surrounding whitespace; trim ttl before comparing by calling
strings.TrimSpace on the input used in the switch (e.g., trim the ttl variable
at the start of the function or just before the switch in the duration
package/function that contains the switch), then perform the existing
comparisons ("0","0s","0m","0h","0d") against the trimmed value so values like "
0s " are recognized as zero TTL.
In `@website/src/data/roadmap.js`:
- Line 297: The progress for the extensibility initiative was decremented
incorrectly when a milestone was marked shipped; update the extensibility
object's progress property (currently `progress: 83`) to reflect the shipped
milestone (e.g., `progress: 85`) or, better, recalculate progress by counting
shipped entries in the extensibility.milestones array divided by total
milestones and writing that computed value back to the extensibility.progress
field; locate the `extensibility` object and its `progress` property in
roadmap.js and either adjust the literal or implement the recalculation logic
that derives progress from `milestones`.
---
Nitpick comments:
In `@website/src/data/roadmap.js`:
- Line 323: Add a pr field linking the PR number to the roadmap entry by
updating the object with label 'Remote import source caching (clone once + TTL)'
to include pr: 2571 (e.g., add pr: 2571 alongside status, quarter, changelog,
docs, description, benefits) so the roadmap row references the originating pull
request for traceability.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4f6bf23e-9434-49bc-9e47-5bee61cef71f
📒 Files selected for processing (13)
internal/exec/stack_processor_utils.gopkg/datafetcher/schema/atmos/manifest/1.0.jsonpkg/datafetcher/schema/config/global/1.0.jsonpkg/datafetcher/schema/stacks/stack-config/1.0.jsonpkg/duration/duration.gopkg/provisioner/source/provision_hook.gopkg/schema/schema.gopkg/stack/imports/remote.gopkg/stack/imports/remote_cache_test.gopkg/stack/imports/remote_nested.gowebsite/blog/2026-06-05-faster-remote-stack-imports.mdxwebsite/docs/stacks/imports.mdxwebsite/src/data/roadmap.js
- duration.IsZeroTTL now trims whitespace before zero-value matching, so " 0s " is recognized as a zero TTL (consistent with Parse/ParseDuration). Add TestIsZeroTTL (with whitespace regression cases) and TestIsExpired. - roadmap: the "Custom hooks with built-in kinds" milestone (#2482) was fully shipped (merged df2c7bd, changelog custom-hooks) but still marked in-progress; flip it to shipped. Extensibility progress recomputed to 26 shipped / 30 total = 87 (was an inaccurate 83). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/duration/duration_test.go (1)
185-213: 💤 Low valueConsider adding an exact boundary test case.
The current tests have comfortable margins (1 minute within 1 hour, 2 hours beyond 1 hour). Adding a case where
updatedAtis exactly at the TTL boundary would explicitly verify the>comparison behavior.📝 Suggested additional test case
{name: "stale entry beyond TTL", updatedAt: now.Add(-2 * time.Hour), ttl: "1h", wantExpired: true}, + {name: "exactly at TTL boundary", updatedAt: now.Add(-1 * time.Hour), ttl: "1h", wantExpired: false}, {name: "invalid TTL returns error", updatedAt: now, ttl: "nonsense", wantErr: true},🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/duration/duration_test.go` around lines 185 - 213, Add a boundary test case in TestIsExpired that checks updatedAt exactly equals the TTL duration: create a test entry named "exact TTL boundary" with updatedAt set to now.Add(-1*time.Hour) and ttl "1h", then call IsExpired and assert the expected boolean for the current implementation (i.e., whether equality is considered expired or not); place this new case in the tests slice alongside the others in the TestIsExpired function to explicitly verify IsExpired's comparison behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pkg/duration/duration_test.go`:
- Around line 185-213: Add a boundary test case in TestIsExpired that checks
updatedAt exactly equals the TTL duration: create a test entry named "exact TTL
boundary" with updatedAt set to now.Add(-1*time.Hour) and ttl "1h", then call
IsExpired and assert the expected boolean for the current implementation (i.e.,
whether equality is considered expired or not); place this new case in the tests
slice alongside the others in the TestIsExpired function to explicitly verify
IsExpired's comparison behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: eeadd4f4-e17c-44a3-bdc7-e547bf68ef78
📒 Files selected for processing (3)
pkg/duration/duration.gopkg/duration/duration_test.gowebsite/src/data/roadmap.js
✅ Files skipped from review due to trivial changes (1)
- website/src/data/roadmap.js
🚧 Files skipped from review as they are similar to previous changes (1)
- pkg/duration/duration.go
Add a warning admonition to the Remote Imports docs: a remote stack import resolves configuration only, not the referenced component's Terraform/Helmfile root module. describe/list/validate work, but plan/apply fails with "component not found" unless the component is materialized via `source:` (JIT source provisioning) or `atmos vendor pull`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ection
The new global `imports` config section renders as `"imports": {}` in
`atmos describe config` JSON output (encoding/json does not honor omitempty
for empty structs, unlike yaml.v3). Regenerate the two affected golden
snapshots so Acceptance Tests pass on all platforms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2571 +/- ##
==========================================
+ Coverage 78.87% 78.91% +0.03%
==========================================
Files 1201 1201
Lines 115958 116016 +58
==========================================
+ Hits 91464 91555 +91
+ Misses 19435 19402 -33
Partials 5059 5059
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
|
💥 This pull request now has conflicts. Could you fix it Erik Osterman (Cloud Posse) (@osterman)? 🙏 |
Resolve conflict in pkg/stack/imports/resolveGitSubdir: main's auth fix (#2568) added broker.EnsureCredentials before detectGitSource, but this branch refactored resolveGitSubdir to delegate cloning to ensureSourceDir. Port the credential-provisioning fix into ensureSourceDir (remote_nested.go), immediately before its detectGitSource call — the spot where the detect now runs — so both the clone-caching feature and the Atmos Pro github/sts broker ordering are preserved. Moved the broker/context imports accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests for the clone-caching paths that were uncovered: - ResolveRemoteImportNested public wrapper, exercised through the global importer with a mock downloader (consuming globalImporterOnce so the injected importer is not overwritten), including within-run source dedup. - readSourceMetadata error/validation paths: missing marker, malformed JSON, legacy marker without a timestamp, and a valid marker. Raises pkg/stack/imports coverage 93.3% -> 94.7% (readSourceMetadata to 100%, ResolveRemoteImportNested 0% -> 80%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Andriy Knysh (aknysh)
approved these changes
Jun 6, 2026
Contributor
|
Tip Atmos Pro
No affected stacks workflow was detected for this pull request.
|
Erik Osterman (Cloud Posse) (osterman)
added a commit
that referenced
this pull request
Jun 6, 2026
Shipped roadmap milestones require both changelog and pr; the merged remote-import-caching entry (from #2571) was missing pr. Flagged by CodeRabbit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
These changes were released in v1.221.0-rc.6. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
what
describe affectedpasses).ttlto reuse the cloned source across runs until it expires: per-import (ttl:in the import map form) and a globalimports.ttldefault inatmos.yaml. With nottl, the source refreshes once per run so mutable refs like?ref=mainstay fresh.ensureSourceDir, add per-session fetch tracking + TTL freshness (timestamp persisted in the.atmos-source-readymarker), and extract a sharedduration.IsExpired/IsZeroTTLthat the source provisioner now reuses.stacks/imports.mdx, add a changelog blog post, and add a roadmap milestone.why
atmos describe affectedwas re-cloning the hub repo once per import (~68–87×/run, ~7–11 min total), and a warmactions/cacheof~/.cache/atmos/stack-imports/was ignored because the subdir path re-cloned unconditionally.ttllets CI reuse the clone across runs (warm cache skips the clone entirely) while keeping mutable refs fresh by default. Shallow clones (depth=1) were already in use — the win is not re-cloning the same repo repeatedly.references
~/.cache/atmos/stack-imports/, honoringXDG_CACHE_HOME).pkg/duration,pkg/provisioner/source).website/blog/2026-06-05-faster-remote-stack-imports.mdxSummary by CodeRabbit
New Features
ttland globalimports.ttlfor optional cross-run caching of remote stack imports.Documentation
Tests