New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when closing a connection while in a Wait() #1638

Closed
bbycroft opened this Issue Jul 17, 2017 · 3 comments

Comments

Projects
None yet
4 participants
@bbycroft

bbycroft commented Jul 17, 2017

Steps to reproduce

Running this demonstration code will cause a crash when the cancellation token is triggered after 3 seconds. In a real system, the CancellationTokenSource would be cancelled as part of a graceful shutdown.

public static void NpgsqlTestWait() {
    Stopwatch sw = Stopwatch.StartNew();
    Action<string> log = s => Console.WriteLine($"[{sw.ElapsedMilliseconds,5}] {s}");

    var connStr = new NpgsqlConnectionStringBuilder {
        Host = "localhost",
        Port = 5432,
        Database = "postgres",
        Username = "postgres",
        Password = "",
        Pooling = false,
    }.ToString();

    using (var conn = new NpgsqlConnection(connStr))
    using (CancellationTokenSource cts = new CancellationTokenSource(TimeSpan.FromSeconds(3))) {
        conn.StateChange += (o, args) => log($"state change: {args.OriginalState} -> {args.CurrentState}");
        conn.Notification += (o, args) => log($"notification: ({args.Condition}) {args.AdditionalInformation}");

        cts.Token.Register(() => {
            log("Cancellation requested; closing connection");
            conn.Close(); // NULL-ref occurs here when Pooling = true
            log("connection closed");
        });

        log("Opening connection");

        conn.Open();

        log("Connection opened");

        while (!cts.IsCancellationRequested) {
            log("Waiting for 10s");

            conn.Wait(TimeSpan.FromSeconds(10)); // NULL-ref occurs here when Pooling = false

            log("Waiting timed out after 10s (or we received a notification)");
        }
    }
    log("Completed");
}

The issue

I want to be able to gracefully shut down the connection in a timely manner, and not have to wait for the timeout to be reached. The only way to do this with a Wait() in progress, afaict, is to call conn.Close(). However, this causes a NullReferenceException in Wait() when Pooling = true or an InvalidOperationException in Close() when Pooling = false.

Exception thrown from conn.Wait() when Pooling = false:

Exception message: System.NullReferenceException: Object reference not set to an instance of an object.
Stack trace:
at Npgsql.NpgsqlConnector.Cleanup()
   at Npgsql.NpgsqlConnector.Break()
   at Npgsql.ReadBuffer.<Ensure>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Npgsql.NpgsqlConnector.<DoReadMessage>d__148.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Npgsql.NpgsqlConnector.<ReadMessage>d__147.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Npgsql.NpgsqlConnector.Wait(Int32 timeout)
   at Npgsql.NpgsqlConnection.Wait(Int32 timeout)
   at Npgsql.NpgsqlConnection.Wait(TimeSpan timeout)
   at ConsoleTesting.NpgsqlWaitTesting.NpgsqlTestWait() in C:\repos\testing\ConsoleTesting\NpgsqlWaitTesting.cs:line 52

Exception thrown from conn.Close() when Pooling = true:


Exception message: System.InvalidOperationException: Reset() called on connector with state Waiting
Stack trace:
   at Npgsql.NpgsqlConnector.Reset()
   at Npgsql.ConnectorPool.Release(NpgsqlConnector connector)
   at Npgsql.NpgsqlConnection.Close(Boolean wasBroken)
   at Npgsql.NpgsqlConnection.Close()
   at ConsoleTesting.NpgsqlWaitTesting.<>c__DisplayClass1_1.<NpgsqlTestWait>b__3() in C:\repos\testing\ConsoleTesting\NpgsqlWaitTesting.cs:line 39

I gather that the error message "Reset() called on connector with state Waiting" is the more accurate message. However, I cannot otherwise close the connection in a timely manner.

Further technical details

Npgsql version: 3.2.4.1
PostgreSQL version: 9.4
Operating system: Windows 10

@Logerfo

This comment has been minimized.

Show comment
Hide comment
@Logerfo

Logerfo Nov 13, 2017

You cannot (currently) close a connection while it is doing something - either processing a command or during Wait(), that will throw the exception you received (Reset() called on connector with state Waiting). This should improve in 3.3 with #1127.

quote by @roji: #1572 (comment)

Logerfo commented Nov 13, 2017

You cannot (currently) close a connection while it is doing something - either processing a command or during Wait(), that will throw the exception you received (Reset() called on connector with state Waiting). This should improve in 3.3 with #1127.

quote by @roji: #1572 (comment)

@cratu

This comment has been minimized.

Show comment
Hide comment
@cratu

cratu Jan 16, 2018

Hello.
I have the same trouble, but I use WaitAsync: when I do cancel via cancelation token and try unsubscribe or close connection I have an exception.
So I use workaround like this:

            source.Cancel(); // source is CancellationTokenSource
            for (int i = 1; i < 50; i++)
            {
                if (!_notificationConnection.FullState.HasFlag(System.Data.ConnectionState.Fetching))
                    break;
                Thread.Sleep(i*10);
            }

This code wait around 20-25 ms on my core i5 for connection release.

cratu commented Jan 16, 2018

Hello.
I have the same trouble, but I use WaitAsync: when I do cancel via cancelation token and try unsubscribe or close connection I have an exception.
So I use workaround like this:

            source.Cancel(); // source is CancellationTokenSource
            for (int i = 1; i < 50; i++)
            {
                if (!_notificationConnection.FullState.HasFlag(System.Data.ConnectionState.Fetching))
                    break;
                Thread.Sleep(i*10);
            }

This code wait around 20-25 ms on my core i5 for connection release.

@roji

This comment has been minimized.

Show comment
Hide comment
@roji

roji Jun 9, 2018

Member

Closing this as a dup of #1127. Doing any operation concurrently with close is currently unsupported and the behavior is undefined.

Member

roji commented Jun 9, 2018

Closing this as a dup of #1127. Doing any operation concurrently with close is currently unsupported and the behavior is undefined.

@roji roji closed this Jun 9, 2018

@roji roji added the duplicate label Jun 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment