[BEAM-10549] Improve runtime type checking performance for the Python SDK #12352

saavan-google · 2020-07-23T06:49:22Z

This PR creates a new flag, performance_runtime_type_check which provides faster runtime type checking by sampling values.

Tests and documentation have been added to reflect this change.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

sdks/python/apache_beam/runners/worker/opcounters.py

saavan-google · 2020-07-23T23:08:45Z

Retest this please

saavan-google · 2020-07-28T02:13:04Z

Retest this please

saavan-google · 2020-08-02T23:28:45Z

R: @udim
R: @robertwb

PTAL

robertwb

This'll be a lot faster .

Most of the comments revolve around pushing the figuring-out-what-to-check logic earlier, so the code in opcounters just becomes a dumb checker. E.g. we should be able to construct the map {"Input to Full/Transform/Label": int, "Output of Another/Map(fn_name)": float} in the visitor and then pass that around. (This should be easier to unit test as well.)

sdks/python/apache_beam/runners/common.py

sdks/python/apache_beam/options/pipeline_options.py

sdks/python/apache_beam/runners/worker/bundle_processor.py

sdks/python/apache_beam/runners/worker/operations.py

sdks/python/apache_beam/typehints/typecheck.py

website/www/site/content/en/documentation/sdks/python-type-safety.md

sdks/python/apache_beam/pipeline.py

sdks/python/apache_beam/runners/worker/opcounters.py

robertwb

As discussed in person, let's push the construction of a {source_str: TypeConstraint} all the way up to the visitor.

sdks/python/apache_beam/runners/worker/bundle_processor.py

saavan-google · 2020-08-06T23:57:46Z

As discussed in person, let's push the construction of a {source_str: TypeConstraint} all the way up to the visitor.

Just pushed these changes - PTAL

I kept get_perf_runtime_type_hints because it's merely responsible for unpickling the function to get the constructed type hints dict

sdks/python/apache_beam/runners/worker/opcounters.py

sdks/python/apache_beam/runners/common.py

sdks/python/apache_beam/runners/worker/opcounters.py

sdks/python/apache_beam/runners/worker/operations.py

sdks/python/apache_beam/typehints/typecheck.py

robertwb · 2020-08-10T18:21:28Z

As discussed in person, let's push the construction of a {source_str: TypeConstraint} all the way up to the visitor.

Just pushed these changes - PTAL

I kept get_perf_runtime_type_hints because it's merely responsible for unpickling the function to get the constructed type hints dict

The problem with this function is that it's called on all sorts of operations, even those without type constraints, when it applies to only one kind of operation. This pattern (and the heavy use of hasattr) suggests it'd be better to make this a method on Operation, say with default of return None.

saavan-google · 2020-08-11T05:50:24Z

The problem with this function is that it's called on all sorts of operations, even those without type constraints, when it applies to only one kind of operation. This pattern (and the heavy use of hasattr) suggests it'd be better to make this a method on Operation, say with default of return None.

That makes sense, just pushed fixes with these design changes.

PTAL @robertwb

robertwb

Sorry I wasn't able to get to this earlier. This is looking really good, only minor comments at this point.

Also, could you run the microbenchmark before and after your change with all typechecking disabled and post the results to verify those timing remain unchanged?

sdks/python/apache_beam/examples/snippets/snippets.py

sdks/python/apache_beam/runners/worker/opcounters.py

sdks/python/apache_beam/runners/worker/operations.py

sdks/python/apache_beam/transforms/ptransform.py

sdks/python/apache_beam/transforms/core.py

sdks/python/apache_beam/typehints/typecheck.py

saavan-google · 2020-08-17T17:08:35Z

PTAL @robertwb

udim · 2020-08-17T22:55:35Z

Run Python PreCommit

robertwb

Just a couple of cosmetic fixes and pending the comparison of benchmarks to master and this'll be ready to merge.

sdks/python/apache_beam/examples/snippets/snippets.py

sdks/python/apache_beam/typehints/typecheck.py

sdks/python/apache_beam/runners/worker/opcounters.py

…isinstance(transform, ParDo) check in the visitor

saavan-google · 2020-08-20T07:15:53Z

run python precommit

saavan-google · 2020-08-20T16:54:15Z

run python precommit

saavan-google · 2020-08-21T03:58:38Z

retest this please

saavan-google · 2020-08-21T04:51:12Z

run python unit tests

robertwb

Thanks! I've reviewed the benchmarks as well; they seem to be in parameters now.

… SDK (apache#12352) * Add performance runtime type check flag * Add PerformanceTypeCheckVisitor which stores type hints in DoFns * Store the type hints in the OperationCounters class * Runtime type checking works with input values * Runtime type checking works with output values * Declare type_check function in cython and reduce error handling code * Throw error if both flags are on * Use producer and consumer Operations' type hints * Store in _runtime_type_hints instead of _type_hints * Simplify extraction process of type hints * Rename variables * Use correct type hints in broken tests * Add equivalent tests for both runtime type checking flags without duplication * Remove accidental changes * Fixup: apply YAPF * Remove redundant test * Fix cython error * Fix cython error * Fix cython error * Fix update_counters_finish * Fix bugs, and py2 incompatability * Fixup: apply YAPF * Fix line-too-long error * For testing purposes, default to performance_runtime_type_check when both flags are on * Fix import order * Fixup: apply YAPF * Fix bug where CombineFns could get type hints * Fix ConsumerSet init bug * Ignore all combine classes, and fix iteration bug * Add full labels to transforms * Provide more descriptive error message * Fix bug where non-iterables wouldn't be type-checked * Store argspec * Compute type hints in visitor instead of opcounters for perf gains, and to get the main input kwarg * Fixup: apply YAPF * Fixup: apply YAPF * Fix import order * Fix lint errors * Use param name in error string * Fixup: apply YAPF * Fix lint * Fix bug where inspect can fail * Check for dict, str, unicode, bytes types in OutputProcessor * Fixup: apply YAPF * Make feature Python 3 only * Update tests to be compatible with both flags * Add tests * Add more tests * Fix lint errors * Add more tests * Leave performance_runtime_type_check off by default * Fix lint * Add website documentation * Use the same value in tests because sampling is non-deterministic * Update a test * Remove accidental changes * PR Fixes * Stage 1 Refactor * Refactor Stage 2 * Refactor Stage 3 * Refactor Stage 4 * Refactor Stage 5 * Fixup: apply YAPF * Add comments for how to type-check side inputs, and lint fixes * Use mega type hint * PR Fixes: fix None bug, empty type hints, and move label transformation to visitor * Fix some PR comments * Add new method to PTransform: _add_type_constraint_from_consumer * Get runtime type hints only from DoOperation instead of all operations * Fix lint errors * Fix lint and cython error * Fix test that was written incorrectly * Add microbenchmark * Fix benchmark & lint errors * Use averaging in benchmark * Remove iterable check in common.py for performance concerns * Add TODO for fixing error message strings * Update documentation strings * Update test and move annotations to a comment * Make logic easier to read and document type hints for _add_type_constraint_from_consumer * Remove ParDo prefix and fix bug * Remove unnecessary try/except and optimize a selector statement * Use type hints for combine functions and add a TODO for removing the isinstance(transform, ParDo) check in the visitor * Extract type hints for all transform types, not just ParDo's * Fix lint errors * Fix type hint annotation docstrings * Add test and fix type hint docstring * Turn the perf RTC flag on for testing purposes * Add try/catch for getfullargspec error * Fix IndexError * Turn flag off * Use cleaner syntax * Rebase onto master * Fix test * Remove unneeded code. * Fix snippets bug * Remove accidental changes * Add TODO * Remove expensive isinstance check * Remove unused import * Fix TODO Co-authored-by: Robert Bradshaw <robertwb@gmail.com>

probot-autolabeler bot added the python label Jul 23, 2020

pabloem reviewed Jul 23, 2020

View reviewed changes

sdks/python/apache_beam/runners/worker/opcounters.py Outdated Show resolved Hide resolved

probot-autolabeler bot added the website label Aug 2, 2020

robertwb reviewed Aug 3, 2020

View reviewed changes

robertwb reviewed Aug 6, 2020

View reviewed changes

sdks/python/apache_beam/runners/worker/bundle_processor.py Outdated Show resolved Hide resolved

saavan-google commented Aug 6, 2020

View reviewed changes

sdks/python/apache_beam/runners/worker/opcounters.py Outdated Show resolved Hide resolved

robertwb requested changes Aug 10, 2020

View reviewed changes

probot-autolabeler bot added the examples label Aug 11, 2020

robertwb reviewed Aug 15, 2020

View reviewed changes

robertwb reviewed Aug 19, 2020

View reviewed changes

sdks/python/apache_beam/examples/snippets/snippets.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/typehints/typecheck.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/runners/worker/opcounters.py Show resolved Hide resolved

saavan-google added 12 commits August 19, 2020 15:28

Add performance runtime type check flag

9cea2f7

Add PerformanceTypeCheckVisitor which stores type hints in DoFns

a1610a8

Store the type hints in the OperationCounters class

4cb5ca1

Runtime type checking works with input values

59daa2d

Runtime type checking works with output values

8c44258

Declare type_check function in cython and reduce error handling code

1f600d9

Throw error if both flags are on

f708d1d

Use producer and consumer Operations' type hints

413d8fe

Store in _runtime_type_hints instead of _type_hints

e9fee8b

Simplify extraction process of type hints

fede55b

Rename variables

645b6c2

Use correct type hints in broken tests

6d0e67d

saavan-google added 13 commits August 19, 2020 15:51

Remove ParDo prefix and fix bug

fd6dcdf

Remove unnecessary try/except and optimize a selector statement

55ea454

Use type hints for combine functions and add a TODO for removing the …

0644a4d

…isinstance(transform, ParDo) check in the visitor

Extract type hints for all transform types, not just ParDo's

f8c9f74

Fix lint errors

cb4aa1c

Fix type hint annotation docstrings

2a6af1c

Add test and fix type hint docstring

8aecf61

Turn the perf RTC flag on for testing purposes

7f6a261

Add try/catch for getfullargspec error

dd12f83

Fix IndexError

dcca2f0

Turn flag off

cfafa0d

Use cleaner syntax

f39c47e

Rebase onto master

68b4ce8

saavan-google force-pushed the BEAM-10549-improve-runtime-type-checking-performance branch from fd1f391 to 68b4ce8 Compare August 19, 2020 20:57

saavan-google and others added 4 commits August 19, 2020 16:40

Fix test

1dc13b0

Remove unneeded code.

e85777a

Fix snippets bug

61f94fd

Remove accidental changes

5e07a9f

saavan-google added 3 commits August 20, 2020 11:55

Add TODO

a23ea8b

Remove expensive isinstance check

c0bd248

Remove unused import

f396eb1

Fix TODO

e9314c3

robertwb approved these changes Aug 21, 2020

View reviewed changes

robertwb merged commit c9bed1e into apache:master Aug 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-10549] Improve runtime type checking performance for the Python SDK #12352

[BEAM-10549] Improve runtime type checking performance for the Python SDK #12352

saavan-google commented Jul 23, 2020 •

edited

Loading

saavan-google commented Jul 23, 2020

saavan-google commented Jul 28, 2020

saavan-google commented Aug 2, 2020

robertwb left a comment

robertwb left a comment

saavan-google commented Aug 6, 2020

robertwb commented Aug 10, 2020

saavan-google commented Aug 11, 2020

robertwb left a comment

saavan-google commented Aug 17, 2020

udim commented Aug 17, 2020

robertwb left a comment

saavan-google commented Aug 20, 2020

saavan-google commented Aug 20, 2020

saavan-google commented Aug 21, 2020

saavan-google commented Aug 21, 2020

robertwb left a comment

[BEAM-10549] Improve runtime type checking performance for the Python SDK #12352

[BEAM-10549] Improve runtime type checking performance for the Python SDK #12352

Conversation

saavan-google commented Jul 23, 2020 • edited Loading

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

saavan-google commented Jul 23, 2020

saavan-google commented Jul 28, 2020

saavan-google commented Aug 2, 2020

robertwb left a comment

Choose a reason for hiding this comment

robertwb left a comment

Choose a reason for hiding this comment

saavan-google commented Aug 6, 2020

robertwb commented Aug 10, 2020

saavan-google commented Aug 11, 2020

robertwb left a comment

Choose a reason for hiding this comment

saavan-google commented Aug 17, 2020

udim commented Aug 17, 2020

robertwb left a comment

Choose a reason for hiding this comment

saavan-google commented Aug 20, 2020

saavan-google commented Aug 20, 2020

saavan-google commented Aug 21, 2020

saavan-google commented Aug 21, 2020

robertwb left a comment

Choose a reason for hiding this comment

saavan-google commented Jul 23, 2020 •

edited

Loading