Skip to content

Init command hangs on database_is_empty() when database already exists on SerenDB/Neon #23

@taariq

Description

@taariq

Problem

When running init against SerenDB/Neon targets where the database already exists, the job hangs indefinitely at the database_is_empty() check. This has been reproduced on multiple jobs:

  • Job 438c3593-057c-45f3-9d23-6ddce6cea1bb
  • Job 314eacf8-5fcc-48d5-822a-3350f3b5a0e3

Both jobs hang for 6+ minutes at the exact same point:

2025-12-03T20:53:37Z INFO: Database 'serendb' already exists on target
[HANG - no further output]

Root Cause Analysis

The issue occurs in src/commands/init.rs at line 403:

  1. Line 377: target_client connects to the target database
  2. Line 388: CREATE DATABASE fails with DUPLICATE_DATABASE error (expected)
  3. Line 403: database_is_empty() is called, which creates a new connection at line 677

The problem is that database_is_empty() creates a second connection while target_client is still holding the first one. SerenDB/Neon has connection pooling limits, and the second connection blocks waiting for a slot.

Additionally, tokio_postgres::connect has no connection timeout, so it hangs indefinitely rather than failing.

Proposed Fix

Change database_is_empty() to accept an existing &Client reference instead of creating a new connection:

// Before (creates new connection)
async fn database_is_empty(target_url: &str, db_name: &str) -> Result<bool> {
    let db_url = replace_database_in_url(target_url, db_name)?;
    let client = postgres::connect_with_retry(&db_url).await?;  // HANG HERE
    // ...
}

// After (reuses existing connection)
async fn database_is_empty(client: &Client) -> Result<bool> {
    let query = "
        SELECT COUNT(*)
        FROM information_schema.tables
        WHERE table_schema NOT IN ('pg_catalog', 'information_schema')
    ";
    let row = client.query_one(query, &[]).await?;
    let count: i64 = row.get(0);
    Ok(count == 0)
}

Then update the call site at line 403 to pass the existing target_client.

Additional Improvement

Consider adding a connection timeout to connect_with_retry() to prevent indefinite hangs in similar scenarios.

Environment

  • SerenDB/Neon targets with channel_binding=require
  • Connection pooling with limited concurrent connections
  • Remote execution via AWS EC2 workers

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions