Default FlowControlSettings LimitExceededBehavior

[JOG-NTMK]

Hi I'm having an issue with my code, I recently started getting
com.google.api.gax.batching.FlowController$FlowControlRuntimeException: The maximum number of batch bytes: 10485760 have been reached.

I can see that's the default behaviour for LoggingOptions.getDefaultInstance().getService(); but the documentation isn't very clear on why its that way or what the alternatives are.
I can see from com.google.api.gax.batching.FlowControlSettings that there are 2 other alternatives Block and Ignore.

I think it would be helpful if somewhere in the documentation these options and their implications were explained and also how to switch between them.

I'd be interested to know if I can switch to Block and what that would mean for my program.

My program is pulling data from an activeMQ queue, logging it to google and then further processing it.
I have to restart the program or else the error continues on indefinitely. Could this error be caused by a message from the queue being above the limit? and if so if set to block will it process the message anyway?

How should this exception be handled so that I can log the data?

Thanks

[pongad]

@garrettjonesgoogle I created googleapis/gax-java#451 to better document the behaviors. @JasonNTMK Please let us know if this answers your question.

Logging does not provide a way for you to change the limit exceeded behavior. Though, I think throwing exception is definitely the wrong thing to do here. I made a separate PR so that the client will throttle instead. Garrett, do you know why throwing exception is the default behavior in autogen clients? It seems...dangerous.

[JOG-NTMK]

It certainly clarifies things for me. Thanks!

One follow question, once the exceptions started it continued until I cleared the queue and any data on it. It happened once more and we didn't log the data that caused the errors but we've since changed to catch the exceptions and log to file so we can see what data is causing the error.
Is it a case of our data being too big to be logged or just sending too much at a time?

It's probably not an easy question to answer without knowing the data but if its too big to be logged (over 100 KB) would blocking permanently hold up as it's never going to be accepted? Our data shouldn't be that big but it could potentially be possible.

I ask because if it was exceeding a sending quota shouldn't it of eventually resolved itself once it was allowed again?

We will have to do some more work our side to check if anythings too big and better handle exceptions but it will help to know the expected behaviour a bit more.

Thanks for your help on this

[pongad]

@JasonNTMK From the issue description, the exception thrown was "The maximum number of batch bytes". This can mean either that you're trying to send too many small logs or some of your logs are large.

It should never be the case that we block permanently. For example, if your byte-limit is 10 bytes but your log is 12 bytes, the flow controller will treat the log is 10 bytes large. It might cause a little "hiccup"; since that one log takes the whole 10 bytes, no other messages can be logged until the big log finishes or fails. It shouldn't hold up permanently though.

[JOG-NTMK]

@pongad Thanks for your help, thats answers all my questions.

[pongad]

Let's keep the issue open until the change lands.