v1.4.0.dev3 — AWS BYOC & Deployment Improvements
Release Date: 2026-07-30
Changes: v1.4.0.dev3 → aws-byoc
Summary
This release adds Bring-Your-Own-Credentials (BYOC) support for AWS deployments and several improvements to data storage, ingestion reliability, and performance. It also includes security hardening, bug fixes, and a few configuration changes for users who deploy Cognee on AWS.
Highlights
- New AWS BYOC (Bring-Your-Own-Credentials) mode so you can store data in your own AWS account.
- S3-backed dataset storage for scalable, durable archives stored in your cloud account.
- Faster and more reliable ingestion for large files and high-throughput workloads.
- Security and deployment improvements: stronger credential handling and clearer configuration for AWS deployments.
Breaking Changes
- AWS deployment configuration changes: if you intend to use AWS BYOC, you must update your deployment configuration to include the new BYOC settings (bucket, role or credentials). Existing managed deployments that do not opt into BYOC are unaffected. Please read the migration guide in the docs before switching a running dataset to S3-backed storage.
- Migration required to move datasets from local storage to S3-backed storage: moving data between storage types is not automatic. Use the provided CLI tools to migrate datasets; follow the migration steps to avoid downtime or inconsistent state.
New Features
- AWS BYOC: You can now configure Cognee to use your own AWS credentials (Bring-Your-Own-Credentials). This option stores datasets, backups, and blobs in an S3 bucket inside your AWS account instead of Cognee's managed infrastructure. Why it matters: you keep full control of data, billing, and compliance (e.g., audit logs, regional controls, retention rules).
- S3-backed dataset storage: Datasets (the collections of documents or files you add to Cognee) can be stored directly in a customer-owned S3 bucket. What it does: large files and long-term data are uploaded to S3 while metadata remains indexed by Cognee. Why it matters: reduces storage costs for large datasets and makes long-term archival and policy enforcement easier for teams.
- Deployment CLI enhancements: New commands and flags in the Cognee CLI to initialize and validate AWS BYOC settings. What it does: helps you create the correct S3 bucket, test the IAM role or credentials, and verify permissions before running production workloads. Why it matters: reduces setup errors and shortens deployment time.
- Automatic multipart upload handling: Large files are automatically split and uploaded to S3 using multipart uploads when using S3-backed storage. What it does: improves reliability and resumes interrupted uploads. Why it matters: large ingest jobs are less likely to fail and are faster overall.
Improvements
- Improved ingestion reliability: better retry logic and resumable uploads for flaky networks, which reduces failed ingests for large files.
- Clearer AWS configuration and logging: deployment logs now show explicit checks (bucket existence, permissions) and guidance on misconfiguration, making troubleshooting faster.
- Indexing improvements: metadata and content indexing has been tuned to reduce duplicated entries and improve search relevance across datasets (a dataset is the collection of documents you add).
- Better UX for dataset management: UI and CLI show dataset storage location (local vs S3) and give clearer guidance when moving data between storage types.
Performance
- Faster large-file ingestion: parallel multipart uploads and reduced memory pressure during indexing result in significantly faster uploads for files over 100MB.
- Lower memory usage during batch indexing: background indexing now stages smaller chunks, reducing peak memory usage during large imports.
Security
- Stronger credential handling for AWS BYOC: Cognee validates role/credential permissions and avoids persisting plaintext secrets in local configuration where possible. Why it matters: reduces the risk of accidental key exposure.
- Transport security enforced for S3 endpoints: uploads and metadata sync use TLS by default and will warn when a non-TLS endpoint is configured.
Bug Fixes
- Fixed dataset deletion edge case where partial S3 artifacts could be left behind when the deletion process was interrupted.
- Resolved an issue where very large files could stall ingestion under certain network conditions; multipart resumes now reliably continue.
- Fixed incorrect storage location shown in the UI for datasets migrated to S3-backed storage.
- Minor CLI help-text and error-message fixes to make onboarding clearer.
Technical Changes
- Refactor of the storage layer to cleanly separate local and S3-backed implementations, paving the way for additional cloud providers in the future.
- Updated several internal dependencies for AWS SDK and upload utilities to newer, more stable versions.
- Added unit and integration tests around S3 upload paths, IAM permission checks, and multipart upload edge cases.
Compatibility
| Component | Supported / Required |
|---|---|
| Python | >=3.10,<3.15 |
| pydantic | >=2.10.5 |
| litellm | >=1.83.7 |
| fastapi | >=0.116.2,<1.0.0 |
| sqlalchemy | >=2.0.39,<3.0.0 |
| lancedb | >=0.24.3,<1.0.0 |
| ladybug | >=0.16.0,<=0.18.2 |
— The Cognee Team · 2026-07-30