BEAM-3361 Increase Go gRPC message size#4270
Closed
wcn3 wants to merge 82 commits intoapache:go-sdkfrom
Closed
Conversation
Increases the buffer for gRPC messages from 4M to 50M.
Contributor
Author
|
r:@herohde |
Contributor
|
Minor comment, otherwise LGTM |
herohde
approved these changes
Dec 15, 2017
| log.Infof(ctx, "Connecting via grpc @ %s ...", endpoint) | ||
|
|
||
| opts := []grpc.DialOption{grpc.WithInsecure(), grpc.WithBlock()} | ||
| opts := []grpc.DialOption{grpc.WithInsecure(), grpc.WithBlock(), grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(50 << 20))} |
Contributor
There was a problem hiding this comment.
Please make this change in the grpcx helper as well (which this code is incidentally replaced with in #4265).
Contributor
Author
There was a problem hiding this comment.
Done. Thanks for taking care of the timeout change in the new code!
Contributor
Author
There was a problem hiding this comment.
Something went sideways with my Git. I'm going to abandon this and try again.
Also add (optional) access to the contexts on serializing and deserializing pipelines.
It's useful in certain contexts to have Maven direct outputs to a non-standard location. By using this property, we can allow for this, while maintaining the default behavior when this property isn't used. The derivation for the default value comes from http://maven.apache.org/pom.html line 32 of the main POM.
This necessitates a dependency on the Core SDK for Coder and WindowedValue.
This is actually unused, and available within java-fn-execution, which is the preferred module in which to store FnApi libraries.
Various worker cleanups.
- Deprecates previous BigtableOptions providing methods.
This expansion to testPrepareJob demonstrates that the returned artifact staging endpoint does write files to a staging location, and those staged bytes can be read back.
This reverts commit a4e4431. The location was set in a profile, which was not the intent.
It's useful in certain contexts to have Maven direct outputs to a non-standard location. By using this property, we can allow for this, while maintaining the default behavior when this property isn't used. This behavior is applied to the base configuration so it can be used in any profile. The derivation for the default value comes from http://maven.apache.org/pom.html line 32 of the main POM.
(I introduced the bug while merging a different PR...) Also adds a test for the combine fn and exposes the fn. Documents the difference between any() and fixedSize().
Example: +import posixpath import StringIO -import posixpath import unittest
beam_runner_api.Environment is the message to use to represent an execution environment or container.
…ed into target command
This is to make them match with the redesign of internal Dataflow intertransform metrics.
This works fine on linux, but fails on Windows machines.
Move to sdks/java/fn-execution. The outbound observers are generally the same on both ends.
This runs a Pipeline via the Portability Framework. This runner is not yet implemented. This change exists to construct the appropriate module.
This brings additional symmetry to the data client and server. Introduce MultiplexingFnDataReceiver to hide the use of non-closeable consumers within the SDK Harness.
* Minor changes to the FnApi Runner to allow Java to run. * Aligned the GBK urn. * Started a logging service, which Java can't run without. * Added some logging that was helpful for debugging.
The code incorrectly sets a local variable from the property, so the value queried in the session code will always be an empty string.
Increases the buffer for gRPC messages from 4M to 50M.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Increases the buffer for gRPC messages from 4M to 50M.