[C++][Go][Java][FlightRPC] Add support for result set expiration #35500

kou · 2023-05-09T03:09:24Z

Describe the enhancement requested

Based on the proposal in https://docs.google.com/document/d/1jhPyPZSOo2iy0LqIJVUs9KWPyFULVFJXTILDfkadx2g/edit# .
See also the discussion thread: https://lists.apache.org/thread/247z3t06mf132nocngc1jkp3oqglz7jp

Currently, it is undefined whether a client can call DoGet more than once. Clients may want to retry requests, and servers may not want to persist a query result forever.

Proposal

Add an expiration time to FlightEndpoint. If present, clients may assume they can retry DoGet requests. Otherwise, clients should avoid retrying DoGet requests.

message FlightEndpoint {
  // In UTC.
  google.protobuf.Timestamp expiration_time = 3;
}

This proposal is not a full retry protocol.

Also, add “pre-defined” actions to Flight RPC for working with result sets. These are pre-defined Protobuf messages with standardized encodings for use with DoAction:

~~CancelQuery~~ CancelFlightInfo: Asynchronously cancel the execution of a distributed query. (Replaces the equivalent Flight SQL action.)
~~RefreshQuery~~ RefreshFlightEndpoint: Request an extension of the expiration of a FlightEndpoint.
~~CloseQuery~~ ~~CloseFlightInfo: Close a FlightInfo so that the server can clean up resources early.~~ (Only CancelFlightInfo will be enough.)

This lets the ADBC/JDBC/ODBC drivers for Flight SQL explicitly manage result set lifetimes. These can be used with Flight SQL as regular actions.

Prior Art

Dremio: job results are cleaned up after a set time.
Google BigQuery Storage: read sessions include an explicit expiration time.
Snowflake: result sets persist for 24 hours.

Component(s)

C++, FlightRPC

The text was updated successfully, but these errors were encountered:

kou · 2023-05-25T07:53:22Z

https://lists.apache.org/thread/71pp95q6yklodm6lfjttswr3slfowdrb

About CloseQuery:

maybe it would be nice to name it 'CloseFlightInfo', to be matched with GetFlightInfo.

I agree with this. I'll use CancelFlightInfo/RefreshFlightInfo/CloseFlightInfo.

kou · 2023-06-06T08:42:39Z

Ah, should RefreshQuery extend a expiration time of a FlightEndpoint? Or all expiration times of FlightInfo?
If the former, we should use RefereshFlightEndpoint, RefreshFlightInfo otherwise.

kou · 2023-06-06T11:54:07Z

RefreshFlightEndpoint will be better because it's difficult that we return a suitable response for refreshing multiple FlightEndpoints. (If one of multiple FlightEndpoints is only failed, what response is suitable?)

lidavidm · 2023-06-06T11:55:21Z

RefreshFlightEndpoint should be more flexible, agreed.

…ation Currently, it is undefined whether a client can call DoGet more than once. Clients may want to retry requests, and servers may not want to persist a query result forever. Add an expiration time to FlightEndpoint. If present, clients may assume they can retry DoGet requests. Otherwise, clients should avoid retrying DoGet requests. This is not a full retry protocol. Also, add "pre-defined" actions to Flight RPC for working with result sets. These are pre-defined Protobuf messages with standardized encodings for use with DoAction: * CancelFlightInfo: Asynchronously cancel the execution of a distributed query. (Replaces the equivalent Flight SQL action.) * RefreshFlightEndpoint: Request an extension of the expiration of a FlightEndpoint. * CloseFlightInfo: Close a FlightInfo so that the server can clean up resources early. This lets the ADBC/JDBC/ODBC drivers for Flight SQL explicitly manage result set lifetimes. These can be used with Flight SQL as regular actions.

kou · 2023-06-19T06:19:33Z

Discussion:
[DISCUSS][Format][Flight] Result set expiration support
https://lists.apache.org/thread/48fqd554gkqrrld8k13l3b8trz5gk7ow

kou · 2023-06-25T02:48:04Z

Changes based on discussion on the mailing list:

https://lists.apache.org/thread/ofw6bydhyhcqqn3t8013gx0k6fbw8lvv

Remove CloseFlightInfo (if nobody objects it)

RefreshFlightEndpoint ->
RenewFlightEndpoint

RenewFlightEndpoint(FlightEndpoint) ->
RenewFlightEndpoint(RenewFlightEndpointRequest)

CancelFlightInfo(FlightInfo) ->
CancelFlightInfo(CancelFlightInfoRequest)

…ation Currently, it is undefined whether a client can call DoGet more than once. Clients may want to retry requests, and servers may not want to persist a query result forever. Add an expiration time to FlightEndpoint. If present, clients may assume they can retry DoGet requests. Otherwise, clients should avoid retrying DoGet requests. This is not a full retry protocol. Also, add "pre-defined" actions to Flight RPC for working with result sets. These are pre-defined Protobuf messages with standardized encodings for use with DoAction: * CancelFlightInfo: Asynchronously cancel the execution of a distributed query. (Replaces the equivalent Flight SQL action.) * RefreshFlightEndpoint: Request an extension of the expiration of a FlightEndpoint. * CloseFlightInfo: Close a FlightInfo so that the server can clean up resources early. This lets the ADBC/JDBC/ODBC drivers for Flight SQL explicitly manage result set lifetimes. These can be used with Flight SQL as regular actions.

kou · 2023-07-03T00:38:56Z

The vote carried: https://lists.apache.org/thread/no26s310qn3v0n5x830d50k598fh0pvr

…ation (#36009) ### Rationale for this change Currently, it is undefined whether a client can call DoGet more than once. Clients may want to retry requests, and servers may not want to persist a query result forever. ### What changes are included in this PR? Add an expiration time to FlightEndpoint. If present, clients may assume they can retry DoGet requests. Otherwise, clients should avoid retrying DoGet requests. This is not a full retry protocol. Also, add "pre-defined" actions to Flight RPC for working with result sets. These are pre-defined Protobuf messages with standardized encodings for use with DoAction: * CancelFlightInfo: Asynchronously cancel the execution of a distributed query. (Replaces the equivalent Flight SQL action.) * RefreshFlightEndpoint: Request an extension of the expiration of a FlightEndpoint. This lets the ADBC/JDBC/ODBC drivers for Flight SQL explicitly manage result set lifetimes. These can be used with Flight SQL as regular actions. #### Backward compatibility Flight SQL's CancelQuery is deprecated by Flight RPC's CancelFlightInfo. But some clients may not be able to migrate to CancelFlightInfo entirely. If a client can assume that a server requires 13.0.0 or later, the client can always use CancelFlightInfo. Otherwise, the client need to use CancelQuery (for old server) and/or CancelFlightInfo (for newer server). A server needs to implement only CancelFlightInfo. Because Flight SQL server libraries provide the default CancelQuery implementation that delegates to CancelFlightInfo. Clients can use both of Flight SQL's CancelQuery and Flight RPC's CancelFLightInfo by this feature. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: #35500 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

… expiration (apache#36009) ### Rationale for this change Currently, it is undefined whether a client can call DoGet more than once. Clients may want to retry requests, and servers may not want to persist a query result forever. ### What changes are included in this PR? Add an expiration time to FlightEndpoint. If present, clients may assume they can retry DoGet requests. Otherwise, clients should avoid retrying DoGet requests. This is not a full retry protocol. Also, add "pre-defined" actions to Flight RPC for working with result sets. These are pre-defined Protobuf messages with standardized encodings for use with DoAction: * CancelFlightInfo: Asynchronously cancel the execution of a distributed query. (Replaces the equivalent Flight SQL action.) * RefreshFlightEndpoint: Request an extension of the expiration of a FlightEndpoint. This lets the ADBC/JDBC/ODBC drivers for Flight SQL explicitly manage result set lifetimes. These can be used with Flight SQL as regular actions. #### Backward compatibility Flight SQL's CancelQuery is deprecated by Flight RPC's CancelFlightInfo. But some clients may not be able to migrate to CancelFlightInfo entirely. If a client can assume that a server requires 13.0.0 or later, the client can always use CancelFlightInfo. Otherwise, the client need to use CancelQuery (for old server) and/or CancelFlightInfo (for newer server). A server needs to implement only CancelFlightInfo. Because Flight SQL server libraries provide the default CancelQuery implementation that delegates to CancelFlightInfo. Clients can use both of Flight SQL's CancelQuery and Flight RPC's CancelFLightInfo by this feature. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: apache#35500 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

kou added the Type: enhancement label May 9, 2023

github-actions bot added Component: C++ Component: FlightRPC labels May 9, 2023

github-actions bot mentioned this issue Jun 9, 2023

GH-35500: [C++][Go][Java][FlightRPC] Add support for result set expiration #36009

Merged

github-actions bot assigned kou Jun 9, 2023

kou changed the title ~~[C++][FlightRPC] Add support for result set expiration~~ [C++][Go][FlightRPC] Add support for result set expiration Jun 19, 2023

kou closed this as completed in #36009 Jul 3, 2023

kou added this to the 13.0.0 milestone Jul 3, 2023

kou changed the title ~~[C++][Go][FlightRPC] Add support for result set expiration~~ [C++][Go][Java][FlightRPC] Add support for result set expiration Jul 3, 2023

kou mentioned this issue Jul 28, 2023

[C++][Go][Java][FlightRPC] Add support for long-running queries #36155

Closed

lidavidm mentioned this issue Jul 31, 2023

[FlightRPC][Python] Expose new Flight features in Python #36954

Open

7 tasks

lidavidm mentioned this issue Jan 10, 2024

go/adbc/driver/flightsql: support new result set expiration apache/arrow-adbc#1452

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[C++][Go][Java][FlightRPC] Add support for result set expiration #35500

[C++][Go][Java][FlightRPC] Add support for result set expiration #35500

kou commented May 9, 2023 •

edited

Loading

kou commented May 25, 2023

kou commented Jun 6, 2023

kou commented Jun 6, 2023

lidavidm commented Jun 6, 2023

kou commented Jun 19, 2023

kou commented Jun 25, 2023

kou commented Jul 3, 2023

[C++][Go][Java][FlightRPC] Add support for result set expiration #35500

[C++][Go][Java][FlightRPC] Add support for result set expiration #35500

Comments

kou commented May 9, 2023 • edited Loading

Describe the enhancement requested

Proposal

Prior Art

Component(s)

kou commented May 25, 2023

kou commented Jun 6, 2023

kou commented Jun 6, 2023

lidavidm commented Jun 6, 2023

kou commented Jun 19, 2023

kou commented Jun 25, 2023

kou commented Jul 3, 2023

kou commented May 9, 2023 •

edited

Loading