[BEAM-11904] Remove PrettyPrint from job definition

[Fokko]

Please add a meaningful description for your change here. I would like to remove the PrettyPrint from the output. This decreases the size of the job definition:

With the PR:

-rw-r--r--  1 fdriesprong  wheel    45M Mar  1 22:27 /tmp/pubsub-to-bigquery.json

With 2.28.0:

-rw-r--r--  1 fdriesprong  wheel    47M Mar  1 22:30 /tmp/pubsub-to-bigquery.json

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment.
• Format the pull request title like [BEAM-11904] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang				---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[dpcollins-google]

R: @pabloem

I don't know if this would affect anything in the dataflow backend? But AFAICT this would result in more compressed JSON and fewer user jobs over the limit.

@Fokko, I don't know that this solves your issue. Even after this change, your job graph size is still over the limit (20 MiB IIUC?) https://cloud.google.com/dataflow/docs/guides/common-errors#job-graph-too-large

It might be good to look at what is in the serialized JSON payload for your job, see if there's any multi-MiB blobs of data that you shouldn't be serializing?

[pabloem]

right - often closures with larger-than-intended context can explode the job graph size

[Fokko]

We're writing data from PubSub to BigQuery, and there are Avro messages that we know upfront, and that we decode from PubSub. I don't see anything huge, but the serialized_source and the serialized_fn are rather large (not big). I think the schema is represented multiple times in the graph, each time it is being passed to the next stage ((de)serializing).

@dpcollins-google Thanks for the quick reply. I'm iterating towards a <20MB job, which is indeed the limit. I noticed that it was pretty printed, so that was a quick win of around 5%.

[pabloem]

Can you try using --experiments=upload_graph? This should upload your graph to GCS instead of an HTTP request

[pabloem]

let me know if that works, and if so, I can get it on stackoverflow / documentation for others

[pabloem]

Run Java PostCommit

[Fokko]

@pabloem Thanks for the pointer. It looks good at first glance. The UI seems to become unresponsive: Graph monitoring data exceeded maximum allowable size. Some monitoring data including counters may be incomplete. But if it works, that would be okay for now. :)

[pabloem]

hm right, because the UI also passes the job graph in an API call...
I guess a good next-step is to change how we include schemas in the graph to have a dictionary of some sort...

[Fokko]

@pabloem Seems like I'm unable to see the stacktrace of the failed test, can you confirm?

[pabloem]

that's correct - though the failed test is broken on the main branch: https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/testReport/

[pabloem]

Run Java PostCommit

[Fokko]

@pabloem Do you think this is something that we can move forward?

[pabloem]

thanks for rebasing! : )

[pabloem]

Run Java PostCommit

[Fokko]

Thanks @pabloem