Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpc: Wait until resources finish cleaning up in Stop() and GracefulStop() #6489

Merged
merged 31 commits into from Oct 31, 2023

Commits on Jul 31, 2023

  1. transport/http2server: wait in Close() until loopyWriter terminated

    When Server.Stop() returns it is not guaranteed that the loopyWriter
    go-routine terminated.
    
    This can lead to a panic or a testing.(*common).logDepth() race
    condition in Go Tests because t.Log is used after or during the testcase
    terminates.
    This can happen when:
    - a GRPC server is started in a Go test,
    - the GRPC logs are forwarded to t.Log,
    - loopyWriter.run logs an error after server.Stop() and the Test
      method returns.
    
    grpc@v1.57.0/internal/leakcheck is unable to detect that the
    loopyWriter go-routine continues to run after server.Stop() because it
    waits up to 10s for go-routines to terminate after a test finishes.
    The loopyWriter returns much faster after Stop() returns.
    
    To make server.Stop() wait until loopyWriter terminated:
    - rename the existing writerDone field, which is only used in tests, to
      loopyWriterDone, the writerDone channel is closed when the loopyWriter
      go-routine exited
    - change http2server.Close to wait until loopyWriterDone is closed
    fho committed Jul 31, 2023
    Copy the full SHA
    ad96f3c View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. Revert "transport/http2server: wait in Close() until loopyWriter term…

    …inated"
    
    This reverts commit ad96f3c.
    fho committed Aug 9, 2023
    Copy the full SHA
    26fd669 View commit details
    Browse the repository at this point in the history
  2. server: Wait in GracefulStop and Stop until loopyWriter goroutine exits

    It was not guaranteed that when Stop() and GracefulStop() returned, the
    loopyWriter go-routines terminated.
    This can lead to a panic or a testing.(*common).logDepth() race
    condition in Go Tests because t.Log is used after or during the testcase
    terminates.
    This can happen when:
    - a GRPC server is started in a Go test,
    - the GRPC logs are forwarded to t.Log,
    - loopyWriter.run logs an error after server.Stop() and the Test
      method returns.
    
    grpc@v1.57.0/internal/leakcheck is unable to detect that the
    loopyWriter go-routine continues to run after server.Stop() because it
    waits up to 10s for go-routines to terminate after a test finishes.
    The loopyWriter returns much faster after Stop() returns.
    
    To make ensure Stop and GracefulStop only return after the loopyWriter
    go-routine terminated:
    - rename http2Server.writerDone to loopyWriterDone, the channel was only
      used in test and is closed when loopyWriter returns
    - wait in http2Server.HandleStreams() until loopyWriterDone was closed
    - wait in Server.Stop() until all connections were removed, this is
      already done in GracefulStop.
      To abort GracefulStop() when Stop() is called a local copy of Server.lis and
      Server.conns was created and both fields were set to nil and then
      s.csv.Broadcast() was called to wake up an eventually sleeping
      GracefulStop() execution. This does not work anymore when we need to
      wait for connections to be removed in Stop() we need to check the
      length of Server.conns.
      Instead a new abortGracefulStop atomicBool is introduced, that is
      set to true to indicate that a GracefulStop() invocation should be
      aborted when Stop() is run.
    fho committed Aug 9, 2023
    Copy the full SHA
    cb7cfa7 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. Copy the full SHA
    7eb46c0 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    5c056bd View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Copy the full SHA
    60700e0 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    f0250c2 View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    e73ad2c View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    17fd57e View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    62a55c3 View commit details
    Browse the repository at this point in the history
  6. server: stopServerWorkers also in GracefulStop

    The call was only done in Stop() and missing in GracefulStop()
    fho committed Aug 28, 2023
    Copy the full SHA
    6143cf7 View commit details
    Browse the repository at this point in the history
  7. server: consolidate shared Stop and GracefulStop code

    Move code that is used in Stop and GracefulStop to helper functions
    instead of duplicating it. This makes it easier to compare Stop() and
    GracefulStop.
    fho committed Aug 28, 2023
    Copy the full SHA
    8a4ebe0 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Copy the full SHA
    8bba089 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Copy the full SHA
    ecd6289 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    c0b97bb View commit details
    Browse the repository at this point in the history
  3. Revert "server: close listeners and wait for serveWG threads before d…

    …raining"
    
    This reverts commit 0264c98ecd860431ac82a4e799cf6282e08e8a04.
    fho committed Oct 20, 2023
    Copy the full SHA
    b55bee0 View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    4b1e3eb View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. server: fix: stop sequence

    - drain transporters after waiting serve handlers to terminate
    - call closeServerTransports only on non graceful shutdowns, to abruptly
      close the connections
    fho committed Oct 26, 2023
    Copy the full SHA
    acc0243 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    8a58769 View commit details
    Browse the repository at this point in the history
  3. tests: add server.Stop() and server.GracefulStop testcases

    Add 2 testcases that ensure that:
    - GracefulStop blocks and waits until ongoing client grpc calls finis,
    - Stop() returns despite ongoing blocking grpc calls and clients receive
      a connection error
    fho committed Oct 26, 2023
    Copy the full SHA
    e85749a View commit details
    Browse the repository at this point in the history
  4. Delete internal/transport/http2_server.go.orig

    The file was added accidentally
    fho committed Oct 26, 2023
    Copy the full SHA
    11254c2 View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    ecfd5d1 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. Copy the full SHA
    3feef89 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    c03201f View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    e7b4971 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Copy the full SHA
    cb549f2 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    8354300 View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    8bc0f7b View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    d976997 View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    702ac45 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Copy the full SHA
    66b367c View commit details
    Browse the repository at this point in the history