Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-3221] Improve documentation in model pipeline protos #16474

Merged
merged 1 commit into from
Jan 11, 2022
Merged

[BEAM-3221] Improve documentation in model pipeline protos #16474

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions model/fn-execution/src/main/proto/beam_fn_api.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,8 +268,11 @@ message ProcessBundleRequest {
// instantiated and executed by the SDK harness.
string process_bundle_descriptor_id = 1;

// A cache token which can be used by an SDK to check for the validity
// of cached elements which have a cache token associated.
// Contains the cache token and also defines the scope of what the token applies to.
//
// See https://s.apache.org/beam-fn-state-api-and-bundle-processing#heading=h.7ghoih5aig5m
// for additional details on how to use the cache token with the State API
// to cache data across bundle boundaries.
message CacheToken {

// A flag to indicate a cache token is valid for all user state.
Expand All @@ -289,12 +292,17 @@ message ProcessBundleRequest {
SideInput side_input = 2;
}

// The cache token identifier which should be globally unique.
// An opaque token used with the StateKey to create a globally unique
// identifier.
bytes token = 10;
}

// (Optional) A list of cache tokens that can be used by an SDK to reuse
// cached data returned by the State API across multiple bundles.
//
// See https://s.apache.org/beam-fn-state-api-and-bundle-processing#heading=h.7ghoih5aig5m
// for additional details on how to use the cache token with the State API
// to cache data across bundle boundaries.
repeated CacheToken cache_tokens = 2;

// (Optional) Elements to be processed with the bundle. Either all or
Expand Down
7 changes: 3 additions & 4 deletions model/pipeline/src/main/proto/beam_runner_api.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,7 +452,7 @@ message StandardSideInputTypes {
// Represents a view over a PCollection<KV<K, V>>.
//
// StateGetRequests performed on this side input must use
// StateKey.IterableSideInput or StateKey.MultimapSideInput.
// StateKey.MultimapKeysSideInput or StateKey.MultimapSideInput.
MULTIMAP = 1 [(beam_urn) = "beam:side_input:multimap:v1"];
}
}
Expand Down Expand Up @@ -1354,9 +1354,8 @@ message SideInput {
// This access pattern defines the SDK harness <-> Runner Harness RPC
// interface for accessing a side input.
//
// The only access pattern intended for Beam, because of its superior
// performance possibilities, is "beam:sideinput:multimap" (or some such
// URN)
// See StandardSideInputTypes for an enumeration of all side input types
// defined.
FunctionSpec access_pattern = 1;

// (Required) The FunctionSpec of the UDF that adapts a particular
Expand Down