[BEAM-10200] Respect profile_memory option and add memory profiler to…

[y1chi]

… sdk worker

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang		---		---		---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[y1chi]

R: @angoenka

[y1chi]

retest this please

[y1chi]

Run Portable_Python PreCommit

[angoenka]

This also change the behavior from periodic heap dump to i think on demand heap dump.
Shall we also summarize how it should be used?

[angoenka]

Let's log the import failure

[y1chi]

This also change the behavior from periodic heap dump to i think on demand heap dump.
Shall we also summarize how it should be used?

Do you mean the deletion of MemoryReporter? I don't think it is used anywhere but we can also keep it if it is useful for users.

[codecov]

Codecov Report

Merging #12562 into master will increase coverage by 47.83%.
The diff coverage is 6.00%.

@@             Coverage Diff             @@
##           master   #12562       +/-   ##
===========================================
+ Coverage   34.47%   82.30%   +47.83%     
===========================================
  Files         684      455      -229     
  Lines       81483    54583    -26900     
  Branches     9180        0     -9180     
===========================================
+ Hits        28090    44927    +16837     
+ Misses      52972     9656    -43316     
+ Partials      421        0      -421     

Impacted Files	Coverage Δ
sdks/python/apache_beam/utils/profiler.py	28.12% <6.00%> (ø)
io/gcp/dicomclient.py
runners/job/__init__.py
dataframe/__init__.py
...pets/transforms/elementwise/withtimestamps_test.py
runners/portability/flink_runner.py
examples/complete/tfidf_test.py
runners/pipeline_context.py
runners/worker/statecache.py
coders/observable_test.py
... and 1130 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e44c3c2...b29fdcc. Read the comment docs.

[angoenka]

If MemoryProfiler is not used then we don't have to keep it around.
LGTM

[angoenka]

Can you please take a look at failing tests.
We can merge it after they are healthy.

[y1chi]

retest this please

[y1chi]

@angoenka I think this PR can be merged