gmail: Error 400: Precondition check failed., failedPrecondition

[profsmallpine]

Howdy,

I've searched wide and far on this Precondition check failed error, and I can only find issues where users fix the underlying problem and this is solved. We are seeing this error happening, which then kicks it back into a queue. Once the job retries, the API call is successful. I'm not sure if it's related to this go client, but I have not been able to make any progress on how/why this is occurring. This is how I'm setting up the servoce:

ctx := context.Background()
token := &oauth2.Token{
	AccessToken:  source.AccessToken,
	RefreshToken: source.RefreshToken,
	Expiry:       time.Unix(source.TokenExpiresAt, 0),
}

service, err := gmail.NewService(ctx, option.WithScopes(gmail.GmailReadonlyScope), option.WithTokenSource(s.Config.TokenSource(ctx, token)))
if err != nil {
	return nil, err
}

message, err := service.Users.Messages.Get(authenticatedUser, email.ExternalID).Do()

I find it strange that when it runs again, it is successful, yet this precondition checked failed is an error that we see often in our logs. Sending out a hail mary in case anyone else is seeing this.

[codyoss]

@profsmallpine Thanks for the report.

1. Can you please tell me which version of google.golang.org/api you are pulling using?
2. Can you please provide the exact error message you are seeing?
3. Does this error only occur once and then a retry fixes it? Or does it come up ever X minutes?

[profsmallpine]

We are using v0.22.0.

The exact error message is googleapi: Error 400: Precondition check failed., failedPrecondition.

This error occurs once and then a retry fixes it. I am attaching a screenshot to give an idea of regularlity.

[codyoss]

@profsmallpine Could you please try upgrading to v0.31.0. There might be something that has sense been fixed in the generated code. Also, a few releases back the level of detail provided in error messages improved. I would be interested to see if we can get more context into what is going on here.

Also, how are you acquiring your access token? Some the the errors seem to be pretty much one hour apart which makes me think that the request might be made with an expired credential, then the next time a request is made on the retry the token has been refreshed.

[profsmallpine]

Yes, I will update the version we are running and continue to monitor.

We are creating the service as we need it and allowing the underlying go http transport is refreshing the token. I don't think the issue lives here, but there could definitely be an edge cause I'm not handling. Given that the job retries a handful of seconds later and succeeds, I'm drawing a blank for where this error is coming from. Hopeful that the error message will be more specific moving forward. I'll post back once I have more information.

ctx := context.Background()
token := &oauth2.Token{
	AccessToken:  source.AccessToken,
	RefreshToken: source.RefreshToken,
	Expiry:       time.Unix(source.TokenExpiresAt, 0),
}

service, err := gmail.NewService(ctx, option.WithScopes(gmail.GmailReadonlyScope), option.WithTokenSource(s.Config.TokenSource(ctx, token)))
if err != nil {
	return nil, err
}

[codyoss]

@profsmallpine Just checking back in, did you find the source of the issue?

[profsmallpine]

After upgrading, we ran into another issue (#665) and had to rollback.

[codyoss]

Were you able to figure out the issue here?

[profsmallpine]

I noticed we were still having the issue after upgrading without any better error messages. However, we had to revert the changes given the transport issues. I am going to circle back to this project in a couple of weeks and see if I can get more information. Thanks for you assistance on this bad boy.

[MrJoy]

To offer some context: I see the same intermittent issue, and I'm using the Ruby client library. So, it's likely that this isn't a bug in the client library, but rather an issue with the internal state of Google's systems. The resolution then would be to encourage developers to retry if they encounter this error.

[wgordon17]

I'm hitting the same in the Python client library