Fix EDGAR ingest idempotency and watermarking - #1319
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (2)
|
| Layer / File(s) | Summary |
|---|---|
store_document runtime-import wrapper etl/ingest_flow.py, etl/edgar_flow.py |
Both files now wrap embeddings.store_document in a runtime import and return 0 if import resolution fails. |
SQLite filing upsert: ON CONFLICT RETURNING etl/ingest_flow.py |
_insert_filing() now uses ON CONFLICT ... DO UPDATE ... RETURNING for SQLite filing identity handling. |
Holdings replace helpers and call sites etl/ingest_flow.py, etl/edgar_flow.py |
Delete-before-insert helpers and transactional replacement paths are added for holdings tied to a filing_id. |
CIK watermark resolution and fetch wiring etl/edgar_flow.py |
_latest_filed_date_for_cik() reads the latest stored filed_date, and edgar_flow() passes that value into fetch_and_store when since is missing. |
Test coverage for replacement and watermark behavior tests/test_edgar_flow.py, tests/test_ingest_flow.py |
Tests cover transaction-backed holdings replacement, duplicate holdings removal, watermark lookup, and the updated fetcher wiring. |
Estimated code review effort
🎯 4 (Complex) | ⏱️ ~45 minutes
Possibly related issues
#1298– The changes implement idempotent holdings replacement and watermark-basedsinceresolution for EDGAR ingest.#1310– The SQLite filing upsert change aligns with the filing identity stability issue described there.
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 2.33% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (4 passed)
| Check name | Status | Explanation |
|---|---|---|
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | ✅ Passed | The title clearly matches the main change: EDGAR ingest idempotency and watermarking. |
| Linked Issues check | ✅ Passed | The changes replace duplicate holdings, add a real EDGAR watermark, and cover both SQLite and Postgres paths as required by #1298. |
| Out of Scope Changes check | ✅ Passed | No clear out-of-scope code changes are evident; the extra lazy import is described as part of the PR's stated objectives. |
✨ Finishing Touches
📝 Generate docstrings
- Create stacked PR
- Commit on current branch
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
codex/issue-1298-edgar-idempotent-watermark
Comment @coderabbitai help to get the list of available commands.
|
Runner dispatch state for codex on PR #1319. Do not edit. |
🤖 Keepalive Loop StatusPR #1319 | Agent: Codex | Iteration 0/12 Current State
Last Codex Run
To retry immediately:
Or wait for the next successful Gate run to automatically retry. Codex output:
🔍 Failure Classification| Error type | infrastructure | 🧠 Task Analysis| Provider | ✅ GitHub Models (primary) |
|
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
etl/edgar_flow.py (1)
333-343: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftMake legacy holdings replacement transactional.
On Postgres, the delete at Line 333 can autocommit before the per-row inserts complete. A failure in
_insert_holding_legacythen leaves the filing with deleted or partially replaced holdings. Wrap the delete and insert loop in one transaction.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@etl/edgar_flow.py` around lines 333 - 343, The legacy holdings replacement in the filing flow is not atomic because `_delete_holdings_for_filing` and the `_insert_holding_legacy` loop can commit separately. Update the logic in `etl/edgar_flow.py` around the holdings replacement path to run the delete and all inserts inside a single transaction on `conn`, so a failure in `_insert_holding_legacy` rolls back the deletion and prevents partially replaced holdings.etl/ingest_flow.py (1)
448-458: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftMake holdings replacement atomic for Postgres.
connect_dbopens Postgres withautocommit=True, so Line 448 can commit the delete before_insert_holdings_rowsfinishes. If parsing/casting/insertion fails mid-loop, the previous holdings are lost or partially replaced. Wrap delete + all inserts in one transaction for the Postgres path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@etl/ingest_flow.py` around lines 448 - 458, The holdings replace flow in the insert path is not atomic for Postgres because `connect_db` uses autocommit, so `_delete_holdings_rows` can commit before `_insert_holdings_rows` completes. Update the holdings replacement logic in `ingest_flow.py` to run the delete and all insert work in a single transaction for the Postgres path, ideally around the code that calls `_delete_holdings_rows` and `_insert_holdings_rows`, so a failure during parsing/casting/insertion rolls back the whole replacement.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@etl/edgar_flow.py`:
- Around line 282-285: The EDGAR watermark query in the filings lookup is too
broad because it computes MAX(filed_date) across all sources for a manager,
which can be advanced by non-EDGAR rows. Update the query in the EDGAR flow’s
watermark logic to filter on the EDGAR source column when selecting the maximum
filed_date, using the existing EDGAR filing query path so only EDGAR rows affect
the watermark.
---
Outside diff comments:
In `@etl/edgar_flow.py`:
- Around line 333-343: The legacy holdings replacement in the filing flow is not
atomic because `_delete_holdings_for_filing` and the `_insert_holding_legacy`
loop can commit separately. Update the logic in `etl/edgar_flow.py` around the
holdings replacement path to run the delete and all inserts inside a single
transaction on `conn`, so a failure in `_insert_holding_legacy` rolls back the
deletion and prevents partially replaced holdings.
In `@etl/ingest_flow.py`:
- Around line 448-458: The holdings replace flow in the insert path is not
atomic for Postgres because `connect_db` uses autocommit, so
`_delete_holdings_rows` can commit before `_insert_holdings_rows` completes.
Update the holdings replacement logic in `ingest_flow.py` to run the delete and
all insert work in a single transaction for the Postgres path, ideally around
the code that calls `_delete_holdings_rows` and `_insert_holdings_rows`, so a
failure during parsing/casting/insertion rolls back the whole replacement.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 7c5d0002-f5fe-4d39-92b7-e9087fd1fc05
📒 Files selected for processing (4)
etl/edgar_flow.pyetl/ingest_flow.pytests/test_edgar_flow.pytests/test_ingest_flow.py
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Python CI / python 3.12
- GitHub Check: Python CI / python 3.13
- GitHub Check: Python CI / lint-ruff
⚠️ CI failures not shown inline (7)
GitHub Actions: Agents Verifier / 2_check.txt: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Agents Verifier / check: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Claude Code Review (Opt-in) / 5_Resolve review target.txt: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Claude Code Review (Opt-in) / Resolve review target: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Health 45 Agents Guard / 0_guard.txt: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Health 45 Agents Guard / guard: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
Commit Status: Gate / gate: Gate / gate
Conclusion: failure
Path classification was cancelled
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Manager-Database repository uses Prefect 2.x - import schedules from prefect.client.schemas.schedules
Files:
tests/test_ingest_flow.pyetl/ingest_flow.pytests/test_edgar_flow.pyetl/edgar_flow.py
+(adapters|api|etl|embeddings.py)/**
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Manager-Database repository has modules with type issues (adapters/, api/, etl/, embeddings.py) - use mypy overrides with ignore_errors for these
Files:
etl/ingest_flow.pyetl/edgar_flow.py
🪛 OpenGrep (1.23.0)
etl/ingest_flow.py
[ERROR] 388-388: SQL query built via f-string passed to execute()/executemany(). Use parameterized queries with placeholders instead.
(coderabbit.sql-injection.python-fstring-execute)
etl/edgar_flow.py
[ERROR] 269-269: SQL query built via f-string passed to execute()/executemany(). Use parameterized queries with placeholders instead.
(coderabbit.sql-injection.python-fstring-execute)
[ERROR] 282-285: SQL query built via f-string passed to execute()/executemany(). Use parameterized queries with placeholders instead.
(coderabbit.sql-injection.python-fstring-execute)
🔇 Additional comments (9)
tests/test_edgar_flow.py (2)
118-120: LGTM!Also applies to: 486-486
423-438: 🎯 Functional CorrectnessNo change needed
_setup_relational_schema(...)already inserts the matchingmanagersrow (manager_id=100,cik="0"), so the watermark tests do exercise the realcik -> manager_idpath.> Likely an incorrect or invalid review comment.tests/test_ingest_flow.py (1)
125-161: LGTM!Also applies to: 367-367
etl/ingest_flow.py (3)
20-38: LGTM!
383-388: LGTM!
264-290: 🎯 Functional CorrectnessNo action needed
etl/edgar_flow.py (3)
20-38: LGTM!
265-269: LGTM!
369-377: LGTM!
Keepalive Work Log (click to expand)
|
|
Closer review fix pushed in Addressed CodeRabbit data-integrity findings:
Validation:
Fresh GitHub checks are pending after the push; next closer pass should re-check #1319 and merge/apply |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_edgar_flow.py`:
- Around line 450-471: The transaction test for _replace_holdings_for_filing()
is too weak because conn.holdings starts empty, so it won’t catch a broken
delete path. Seed a stale holding for filing_id=9001 before calling
flow._replace_holdings_for_filing(), and tighten the assertions to verify the
DELETE statement and its bound parameter(s) in StrictPostgresConnection. Keep
the existing checks on conn.transactions and final holdings so the test proves
replacement behavior through the Postgres path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: c2deb4dd-7167-430d-aed5-34ff56fc31b9
📒 Files selected for processing (4)
etl/edgar_flow.pyetl/ingest_flow.pytests/test_edgar_flow.pytests/test_ingest_flow.py
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Python CI / python 3.12
- GitHub Check: Python CI / python 3.13
⚠️ CI failures not shown inline (4)
GitHub Actions: Health 45 Agents Guard / 0_guard.txt: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Health 45 Agents Guard / guard: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Claude Code Review (Opt-in) / Resolve review target: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
GitHub Actions: Claude Code Review (Opt-in) / 5_Resolve review target.txt: Fix EDGAR ingest idempotency and watermarking
Conclusion: failure
##[group]Run set -euo pipefail
�[36;1mset -euo pipefail�[0m
�[36;1m�[0m
�[36;1m# Use dedicated scripts dir, create if needed�[0m
�[36;1mINSTALL_DIR=""�[0m
�[36;1mif [ -z "$INSTALL_DIR" ]; then�[0m
�[36;1m INSTALL_DIR="$***GITHUB_WORKSPACE***/.github/scripts"�[0m
�[36;1mfi�[0m
�[36;1mmkdir -p "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mecho "📦 Installing `@octokit` dependencies in $INSTALL_DIR..."�[0m
�[36;1mcd "$INSTALL_DIR"�[0m
�[36;1m�[0m
�[36;1mdeclare -a VENDORED_ALIAS_DIRS=()�[0m
�[36;1m�[0m
�[36;1mcleanup_vendor_aliases() ***�[0m
�[36;1m if [ "$***`#VENDORED_ALIAS_DIRS`[@]***" -eq 0 ]; then�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m local cleanup_dir="$INSTALL_DIR"�[0m
�[36;1m if [ -z "$cleanup_dir" ]; then�[0m
�[36;1m echo "::warning::Install dir is empty, skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m if ! pushd "$cleanup_dir" >/dev/null 2>&1; then�[0m
�[36;1m echo "::warning::Failed to enter install dir \"$cleanup_dir\"; skipping vendored alias cleanup"�[0m
�[36;1m return 0�[0m
�[36;1m fi�[0m
�[36;1m�[0m
�[36;1m for vendored_alias in "$***VENDORED_ALIAS_DIRS[@]***"; do�[0m
�[36;1m if [ -z "$vendored_alias" ]; then�[0m
�[36;1m continue�[0m
�[36;1m fi�[0m
�[36;1m rm -rf -- "$vendored_alias" || true�[0m
�[36;1m local parent_dir�[0m
�[36;1m parent_dir=$(dirname "$vendored_alias")�[0m
�[36;1m # Remove empty parent directories that may have been created for scoped packages�[0m
�[36;1m while [ "$parent_dir" != "." ] && [ "$parent_dir" != "/" ]; do�[0m
�[36;1m rmdir -- "$parent_dir" 2>/dev/null || break�[0m
�[36;1m parent_dir=$(dirname "$parent_dir")�[0m
�[36;1m done�[0m
�[36;1m done�[0m
�[36;1m�[0m
�[36;1m popd >/dev/null 2>&1 || true�[0m
�[36;1m***�[0m
�[36;1m�[0m
�[36;1mtrap cleanup_vendor_aliases EXIT�[0m
�[36;1m�[0m
�[36;1mcreate_vendor_aliases() ***�[0m
�[36;1m if [ ! -f "package.json" ];...
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Manager-Database repository uses Prefect 2.x - import schedules from prefect.client.schemas.schedules
Files:
tests/test_ingest_flow.pyetl/ingest_flow.pytests/test_edgar_flow.pyetl/edgar_flow.py
+(adapters|api|etl|embeddings.py)/**
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Manager-Database repository has modules with type issues (adapters/, api/, etl/, embeddings.py) - use mypy overrides with ignore_errors for these
Files:
etl/ingest_flow.pyetl/edgar_flow.py
🔇 Additional comments (5)
etl/ingest_flow.py (2)
19-38: LGTM!Also applies to: 391-423, 483-483
264-290: 🎯 Functional CorrectnessSQLite
RETURNINGneeds a 3.35+ runtime
This path relies onINSERT ... ON CONFLICT ... DO UPDATE ... RETURNING, so the deployed SQLite version must be 3.35 or newer.etl/edgar_flow.py (1)
19-38: LGTM!Also applies to: 272-303, 316-323, 372-380, 406-408
tests/test_edgar_flow.py (1)
5-5: LGTM!Also applies to: 97-127, 408-447
tests/test_ingest_flow.py (1)
3-3: LGTM!Also applies to: 91-105, 271-296
|
Closer review-thread fix pushed in Addressed the remaining CodeRabbit test-strength thread by seeding a stale holding for Validation:
|
Provider Comparison ReportProvider Summary
📋 Full Provider Details (click to expand)openai
anthropic
Agreement
DisagreementNo major disagreements detected. Unique Insights
🔍 LangSmith Traces |
|
Workflow state fingerprint for Agents Verifier. Do not edit. |
Closes #1298
Summary
Validation
Summary by CodeRabbit