Profile and improve performance for the local FnApiRunner

[damccorm]

The FnApiRunner is undergoing a series of changes to support streaming. These changes are altering its execution significantly, and may introduce inefficiencies.

This project has the following deliverables:

• A report with results from profiling the execution of a pipeline, and finding hotspots, and inefficiencies
• Code improvements to speed up the execution of the FnApiRunner
• Improvements to the FnApiRunner manual to instruct others on how to do profiling.

 

Tools that you may need to use:

• Benchmarks for the Python direct runner in http://metrics.beam.apache.org
• The FnApiRunner and TestStream microbenchmarks in sdks/python/apache_beam/tools/
• A profiling tool like: https://pypi.org/project/flameprof/ or https://github.com/uber-archive/pyflame 
• Google Docs

Contact  Pablo in [dev@beam.apache.org|mailto:dev@beam.apache.org] to ask questions about this project.

Imported from Jira BEAM-11869. Original Jira may contain additional context.
Reported by: pabloem.
Subtask of issue #21987