Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flow run using VertexRun fails with a protobuf error when using the timeout scheduling argument #5548

Closed
marvin-robot opened this issue Mar 14, 2022 · 1 comment

Comments

@marvin-robot
Copy link
Member

Opened from the Prefect Public Slack Community

dekel.r: Hey,
I have a flow running using Vertex Run - this is my run config -

flow.run_config = VertexRun(scheduling={'timeout': '3600s'},
                            machine_type='n2-highcpu-80', labels=["ml"],
                            service_account=PREFECT_SERVICRE_ACCOUNT)

It works without the scheduling parameter - I added it and used this documentation -
https://docs.prefect.io/orchestration/flow_config/run_configs.html#vertexrun
Now when registering to Prefect cloud and running I get this error -

Parameter to MergeFrom() must be instance of same class: expected google.protobuf.Duration got str.

Am I missing something?
Thanks

anna: It looks like perhaps the google Python package changed something so that now it no longer expects this timeout as a string but rather as google.protobuf.Duration? But this would be surprising since based on https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration|this:

The JSON representation for Duration is a String that ends in s to indicate seconds and is preceded by the number of seconds, with nanoseconds expressed as fractional seconds.

All we do with this, it seems, on the agent side is passing it to the Vertex task definition in this JSON format https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/vertex/agent.py#L170-L171

anna: Let me open an issue for this since I don't know why it occurs and this would need a deeper investigation:

<@ULVA73B9P> open "Flow run using VertexRun fails with a protobuf error when using the timeout scheduling argument"

Original thread can be found here.

@josephdviviano
Copy link

Did this get resolved? I have the same issue.

@cicdw cicdw closed this as completed Aug 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants