[C++][FlightSQL] Add session management primitives #34865

indigophox · 2023-04-03T17:20:44Z

Describe the enhancement requested

In order to support Flight SQL server session persistence within a server application, we are introducing DoAction messages to support handling setting and checking session parameters, as well as closing sessions (housekeeping).

Additionally, a ServerSessionMiddleware has been provided as an example or utility implementation of handling session persistence via an opaque cookie.

(Note that updates to Location have been removed as it was impossible to design them to be consistent and elegant, so clients (and *DBC drivers) will simply set the desired session options independent of a Location URI.)

Component(s)

C++, FlightRPC

The text was updated successfully, but these errors were encountered:

Relates to DX-64185

…4817) ### Rationale for this change Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling. A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context. ### What changes are included in this PR? "Session" set/get/close Actions and server-side helper middleware. ### Are these changes tested? Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included. ### Are there any user-facing changes? Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions. * Closes: #34865 Lead-authored-by: Paul Nienaber <paul.nienaber@dremio.com> Co-authored-by: Paul Nienaber <github@phox.ca> Co-authored-by: James Duong <jduong@dremio.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: David Li <li.davidm96@gmail.com>

lidavidm · 2024-02-20T13:52:20Z

#40155 to add this to Golang; apache/arrow-adbc#1557 to add this to the client.

build fixes

…es (apache#34817) ### Rationale for this change Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling. A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context. ### What changes are included in this PR? "Session" set/get/close Actions and server-side helper middleware. ### Are these changes tested? Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included. ### Are there any user-facing changes? Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions. * Closes: apache#34865 Lead-authored-by: Paul Nienaber <paul.nienaber@dremio.com> Co-authored-by: Paul Nienaber <github@phox.ca> Co-authored-by: James Duong <jduong@dremio.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: David Li <li.davidm96@gmail.com> # Conflicts: # java/flight/flight-integration-tests/src/main/java/org/apache/arrow/flight/integration/tests/Scenarios.java # java/flight/flight-integration-tests/src/test/java/org/apache/arrow/flight/integration/tests/IntegrationTest.java # testing

…es (apache#34817) ### Rationale for this change Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling. A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context. ### What changes are included in this PR? "Session" set/get/close Actions and server-side helper middleware. ### Are these changes tested? Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included. ### Are there any user-facing changes? Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions. * Closes: apache#34865 Lead-authored-by: Paul Nienaber <paul.nienaber@dremio.com> Co-authored-by: Paul Nienaber <github@phox.ca> Co-authored-by: James Duong <jduong@dremio.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: David Li <li.davidm96@gmail.com>

…0284) ### Rationale for this change Brings Go implementation in parity with recent session management additions to Java and C++: #34865 ### What changes are included in this PR? - Go Flight/FlightSQL implementations of session management RPC handlers - Implementation of cookie-based session middleware - Implementation of stateful (id-lookup based) sessions/tokens - Implementation of stateless (fully encoded) sessions/tokens - Fix minor C++ logic bug when closing sessions - Update Java integration test server to return an empty session if `getSessionOptions` is called before `setSessionOptions` - Refactor of `DoAction` handlers to consolidate the code that is essentially copied between them. - As part of this I found an issue with `CancelFlightInfo` where a copy of the message was being returned instead of a pointer as is typically the case with `proto.Message`'s. I updated the return type and any usage throughout the code base as part of the refactor. ### Are these changes tested? Yes, both integration and unit tests are included. A few tests were added in the Go integration suite beyond the existing coverage in the Java/C++ suites. These tests aim to demonstrate my understanding of session semantics in those scenarios, please let me know if you believe the details are not accurate. Some of the new integration tests failed in the Java/C++ scenarios. I made very minor changes to those implementations to fix certain failures but there are still some remaining bugs (assuming these are testing the right semantics). Specifically: - The integration test for reopening a previously closed session passes on Go/Java, but fails for C++ so it is commented out. - This implementation prefers to set any cookies in the gRPC trailer which works fine for Go/C++, but not for Java. As a temporary workaround this implementation will _also_ set the cookie in the gRPC header if a new session was created. This is sufficient to maintain compatibility with Java stateful sessions where the session ID token can be known at the time of creation, but is not robust to other scenarios such as stateless sessions where in many cases the token cannot be known until after the RPC has completed. ### Are there any user-facing changes? Yes, session management RPC as well as middleware implementations are included. Functionality is entirely additive * GitHub Issue: #40155 Authored-by: joel <joellubi@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>

…es (apache#34817) ### Rationale for this change Flight presently contains no formal mechanism for managing connection/query configuration options; instead, request headers and/or non-query SQL statements are often used in lieu, with unnecessary overhead and poor failure handling. A stateless (from Flight's perspective) Flight format extension is desirable to close this gap for server implementations that use/want connection state/context. ### What changes are included in this PR? "Session" set/get/close Actions and server-side helper middleware. ### Are these changes tested? Integration tests (C++ currently broken due to middleware-related framework issue) and some complex-case unit testing are included. ### Are there any user-facing changes? Non-breaking extensions to wire format and corresponding client/server Flight RPC API extensions. * Closes: apache#34865 Lead-authored-by: Paul Nienaber <paul.nienaber@dremio.com> Co-authored-by: Paul Nienaber <github@phox.ca> Co-authored-by: James Duong <jduong@dremio.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: David Li <li.davidm96@gmail.com>

…nt (apache#40284) ### Rationale for this change Brings Go implementation in parity with recent session management additions to Java and C++: apache#34865 ### What changes are included in this PR? - Go Flight/FlightSQL implementations of session management RPC handlers - Implementation of cookie-based session middleware - Implementation of stateful (id-lookup based) sessions/tokens - Implementation of stateless (fully encoded) sessions/tokens - Fix minor C++ logic bug when closing sessions - Update Java integration test server to return an empty session if `getSessionOptions` is called before `setSessionOptions` - Refactor of `DoAction` handlers to consolidate the code that is essentially copied between them. - As part of this I found an issue with `CancelFlightInfo` where a copy of the message was being returned instead of a pointer as is typically the case with `proto.Message`'s. I updated the return type and any usage throughout the code base as part of the refactor. ### Are these changes tested? Yes, both integration and unit tests are included. A few tests were added in the Go integration suite beyond the existing coverage in the Java/C++ suites. These tests aim to demonstrate my understanding of session semantics in those scenarios, please let me know if you believe the details are not accurate. Some of the new integration tests failed in the Java/C++ scenarios. I made very minor changes to those implementations to fix certain failures but there are still some remaining bugs (assuming these are testing the right semantics). Specifically: - The integration test for reopening a previously closed session passes on Go/Java, but fails for C++ so it is commented out. - This implementation prefers to set any cookies in the gRPC trailer which works fine for Go/C++, but not for Java. As a temporary workaround this implementation will _also_ set the cookie in the gRPC header if a new session was created. This is sufficient to maintain compatibility with Java stateful sessions where the session ID token can be known at the time of creation, but is not robust to other scenarios such as stateless sessions where in many cases the token cannot be known until after the RPC has completed. ### Are there any user-facing changes? Yes, session management RPC as well as middleware implementations are included. Functionality is entirely additive * GitHub Issue: apache#40155 Authored-by: joel <joellubi@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>

indigophox added the Type: enhancement label Apr 3, 2023

github-actions bot added Component: C++ Component: FlightRPC labels Apr 3, 2023

github-actions bot mentioned this issue Apr 3, 2023

GH-34865: [C++][Java][Flight RPC] Add Session management messages #34817

Merged

github-actions bot assigned indigophox Apr 3, 2023

indigophox changed the title ~~Add Session management primitives and Location path parsing~~ Add Flight SQL Session management primitives Jun 28, 2023

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Aug 18, 2023

DX-61034: Arrow changes from apacheGH-34865 PR

65e62d6

Relates to DX-64185

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Sep 19, 2023

DX-61034: Arrow changes from apacheGH-34865 PR

2a03f16

Relates to DX-64185

kou changed the title ~~Add Flight SQL Session management primitives~~ [C++][FlightSQL] Add session management primitives Oct 25, 2023

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 19, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

0799cda

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 19, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

31cf166

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 22, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

445a0b1

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 22, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

46bb9ab

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 23, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

49715ee

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 23, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

a959f36

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 23, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

6642933

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 24, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

2b478fe

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 24, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

872a644

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 24, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

5b85167

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Jan 26, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

0d59ab1

lidavidm closed this as completed in #34817 Feb 20, 2024

lidavidm added this to the 16.0.0 milestone Feb 20, 2024

This was referenced Feb 20, 2024

[Go][FlightRPC] Implement session management #40155

Closed

go/adbc/driver/flightsql: add session management apache/arrow-adbc#1557

Closed

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 20, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

14c0660

build fixes

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 21, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

ae24499

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

4515fac

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024

X-61034: Arrow changes from apacheGH-34865 PR

2e261fb

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024

X-61034: Arrow changes from apacheGH-34865 PR

f8e7ed8

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Feb 26, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

810c41a

joellubi mentioned this issue Feb 28, 2024

GH-40155: [Go][FlightRPC][FlightSQL] Implement Session Management #40284

Merged

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Mar 8, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

43446d2

stevelorddremio added a commit to stevelorddremio/arrow that referenced this issue Mar 11, 2024

DX-61034: Arrow changes from apacheGH-34865 PR

b6b322d

devinrsmith mentioned this issue Oct 24, 2024

Implement Flight CloseSession deephaven/deephaven-core#6284

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[C++][FlightSQL] Add session management primitives #34865

[C++][FlightSQL] Add session management primitives #34865

indigophox commented Apr 3, 2023 •

edited by kou

Loading

lidavidm commented Feb 20, 2024 •

edited

Loading

[C++][FlightSQL] Add session management primitives #34865

[C++][FlightSQL] Add session management primitives #34865

Comments

indigophox commented Apr 3, 2023 • edited by kou Loading

Describe the enhancement requested

See also

Component(s)

lidavidm commented Feb 20, 2024 • edited Loading

indigophox commented Apr 3, 2023 •

edited by kou

Loading

lidavidm commented Feb 20, 2024 •

edited

Loading