Database sync worker not detecting rclone bisync changes

[groksrc]

Problem

After using bm project bisync, files sync correctly to S3 storage but don't appear in the web app at app.basicmemory.com. The web app shows stale data while the actual disk and S3 storage have the correct data.

Root Cause

The sync worker uses incremental scanning based on filesystem modification timestamps to detect changes. However, rclone bisync writes directly to S3 without updating filesystem timestamps in a way the scanner can detect.

Evidence

Logfire logs for tenant 43eb3b88-62ac-26e3-4fc3-ca352617725e show:

• At 2025-10-31T03:18:21.948150Z: Worker detected 2 deletions
• Every 30 seconds after: Worker reports "total_changes=0, found 0 changes (new=0, modified=0, deleted=0, moves=0)"
• New files created via bisync: Never detected

Example log entry:

"Completed incremental scan for directory /app/data/bmem in 111ms, found 0 changes (new=0, modified=0, deleted=0, moves=0)"

Impact

• ❌ Users see incorrect data in web app after using bisync
• ❌ No manual way to force database sync
• ❌ No status endpoint to check sync progress
• ⚠️ The bisync output says "Database sync initiated" but this is misleading - the sync runs but finds nothing to update

Technical Details

• S3 Storage: ✅ Correct (matches local disk)
• PostgreSQL Database: ❌ Stale (shows old deleted files, missing new files)
• Sync Worker: Runs every 30 seconds in sync_worker.py:sync_project_background()
• Detection Method: Incremental scan via filesystem modification times
• The Gap: Rclone → S3 writes don't trigger detectable filesystem changes

Test Case to Reproduce

1. Set up project with bm project sync-setup <project> <path>
2. Run bm project bisync --name <project> --resync
3. Create a new file locally: echo "test" > ~/bmem/tests/newfile.md
4. Run bm project bisync --name <project>
5. Verify file exists in S3: bm project ls --name <project> tests/ ✅ Shows file
6. Check web app at app.basicmemory.com ❌ File not listed
7. Check Logfire sync worker logs: Shows total_changes=0

Suggested Solutions

1. Option A: Trigger full scan after bisync completes (instead of incremental)
2. Option B: Have bisync call a database sync API endpoint that forces a rescan
3. Option C: Make rclone update filesystem metadata in a detectable way
4. Option D: Use S3 event notifications to trigger database sync

Related

• PR feat: SPEC-20 Simplified Project-Scoped Rclone Sync #405 (SPEC-20: Simplified Project-Scoped Rclone Sync)
• src/basic_memory_cloud_api/sync_worker.py
• src/basic_memory/cli/commands/project.py

Severity

High - This breaks the core value proposition of cloud sync. Users can't trust that the web app reflects their actual data after using bisync.

[claude]

Claude finished @groksrc's task —— View job

---

Analysis Complete - Issue #407

Todo List:

• Triage: Add appropriate labels
• Understand the codebase architecture
• Analyze the issue and root cause
• Evaluate solution options
• Provide detailed recommendation

---

Problem Summary

After bm cloud bisync, files sync correctly to S3 but don't appear in the web app at app.basicmemory.com.

Root Cause:

1. rclone bisync writes directly to S3
2. Cloud container filesystem gets files via rclone mount/sync
3. SyncService.scan() uses incremental scanning based on last_scan_timestamp
4. Incremental scan only detects files with mtime > last_scan_timestamp
5. Files synced via rclone may not have detectable timestamp changes
6. Result: Database never gets updated with new files

Architecture Clarification

✅ This repo DOES contain the cloud API!

The Dockerfile shows the cloud deployment runs: basic-memory mcp --transport sse --host 0.0.0.0

This starts the FastAPI server with all the /project/sync endpoints that the issue mentions. So the complete fix CAN be implemented in this repository.

Recommended Solution: Option B (API Enhancement)

Add a force_full parameter to the existing sync endpoint to allow forcing full scans.

Why this is better than Option A:

• ✅ Cleaner API design
• ✅ No separate watermark-clear endpoint needed
• ✅ Works for both CLI bisync AND future use cases
• ✅ Minimal code changes
• ✅ Backward compatible (optional parameter)

Implementation Plan

Step 1: Modify /project/sync endpoint (src/basic_memory/api/routers/project_router.py)

@project_router.post("/sync")
async def sync_project(
    background_tasks: BackgroundTasks,
    sync_service: SyncServiceDep,
    project_config: ProjectConfigDep,
    force_full: bool = Query(False, description="Force full scan instead of incremental"),
):
    """Force project filesystem sync to database."""
    
    # Clear watermark if force_full requested
    if force_full:
        await clear_scan_watermark(sync_service)
    
    background_tasks.add_task(sync_service.sync, project_config.home, project_config.name)
    logger.info(f"Filesystem sync initiated for project: {project_config.name}")
    
    return {
        "status": "sync_started",
        "message": f"Filesystem sync initiated for project '{project_config.name}'",
    }

Step 2: Add helper function (in project_router.py)

async def clear_scan_watermark(sync_service: SyncServiceDep):
    """Clear scan watermark to force full scan on next sync."""
    if sync_service.entity_repository.project_id is not None:
        project = await sync_service.project_repository.find_by_id(
            sync_service.entity_repository.project_id
        )
        if project:
            await sync_service.project_repository.update(
                project.id,
                {
                    "last_scan_timestamp": None,
                    "last_file_count": None,
                },
            )

Step 3: Update notify_container_sync() (src/basic_memory/cli/commands/cloud/bisync_commands.py)

async def notify_container_sync(tenant_id: str) -> None:
    """Sync all projects after bisync completes."""
    try:
        from basic_memory.cli.commands.command_utils import run_sync
        
        cloud_data = await fetch_cloud_projects()
        projects = cloud_data.projects
        
        if not projects:
            console.print("[dim]No projects to sync[/dim]")
            return
        
        console.print(f"[blue]Triggering full database sync for {len(projects)} project(s)...[/blue]")
        
        for project in projects:
            project_name = project.name
            if project_name:
                try:
                    # Force full scan after bisync
                    await run_sync(project=project_name, force_full=True)
                except Exception as e:
                    console.print(f"[yellow]  ⚠ Sync failed for {project_name}: {e}[/yellow]")
        
        console.print("[dim]Note: Database indexing started and may take a few moments[/dim]")
    except Exception as e:
        console.print(f"[yellow]Warning: Post-sync notification failed: {e}[/yellow]")

Step 4: Update run_sync() helper (src/basic_memory/cli/commands/command_utils.py)

async def run_sync(project: Optional[str] = None, force_full: bool = False):
    """Run sync operation via API endpoint."""
    try:
        async with get_client() as client:
            project_item = await get_active_project(client, project, None)
            url = f"{project_item.project_url}/project/sync"
            if force_full:
                url += "?force_full=true"
            response = await call_post(client, url)
            data = response.json()
            console.print(f"[green]✓ {data['message']}[/green]")
    except (ToolError, ValueError) as e:
        console.print(f"[red]✗ Sync failed: {e}[/red]")
        raise typer.Exit(1)

Implementation Complexity

Effort: Simple - ~40 lines of code across 3 files

• Modify project_router.py: ~20 lines
• Modify bisync_commands.py: ~5 lines
• Modify command_utils.py: ~5 lines
• Add tests: ~10 lines

Testing Strategy

1. Unit test: Test sync endpoint with force_full=true parameter
2. Integration test: Test bisync → notify → full scan flow
3. Manual test: Run bisync and verify files appear in web app

Benefits

✅ Solves the immediate issue (bisync detection)
✅ Provides manual override for users (if needed)
✅ Clean, maintainable API design
✅ Backward compatible
✅ Reusable for other scenarios (e.g., after manual S3 uploads)

---

Would you like me to implement this solution?

I can create the PR with:

• Code changes to all 3 files
• Tests for the new functionality
• Update to documentation if needed