From d01120b14d6e6ae91096e5884fad32992fceec56 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 12:58:38 -0500 Subject: [PATCH 1/7] Improve explanations in reports. --- filprofiler/_report.py | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/filprofiler/_report.py b/filprofiler/_report.py index bcf76a72..6bef2b98 100644 --- a/filprofiler/_report.py +++ b/filprofiler/_report.py @@ -97,19 +97,40 @@ def render_report(output_path: str, now: datetime) -> str: in the alpha program.
+

Now what?

+ +

Need help reducing your data processing application's memory use? Check out tips and tricks here.

+

Understanding the graphs

The flame graphs shows the callstacks responsible for allocations at peak.

The wider (and the redder) the bar, the more memory was allocated by that function or its callers. If the bar is 100% of width, that's all the allocated memory.

+

The left-right axis has no meaning! +The order of frames is somewhat arbitrary, for example beause multiple calls to the same function may well have been merged into a single callstack. +So you can't tell from the graph which allocations happened first. +All you're getting is that at peak allocation these time, these stacktraces were responsible for these allocations. +

+

The first graph shows the normal callgraph: if main() calls g() calls f(), let's say, then main() will be at the top. The second graph shows the reverse callgraph, from f() upwards.

Why is the second graph useful? If f() is called from multiple places, in the first graph it will show up multiple times, at the bottom. In the second reversed graph all calls to f() will be merged together.

-

Need help reducing your data processing application's memory use? Check out tips and tricks here.

+

Understanding what Fil tracks

+ +

Fil measures how much memory has been allocated; this is not the same as how much memory the process is actively using, nor is it the same as memory resident in RAM.

+ + + +

See this article for more details.

+ """.format( From b81d1f167948bea6a9c9ec1ef21a7bdea7eb984b Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 13:00:11 -0500 Subject: [PATCH 2/7] News file. --- .changelog/185.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 .changelog/185.doc diff --git a/.changelog/185.doc b/.changelog/185.doc new file mode 100644 index 00000000..823bfebf --- /dev/null +++ b/.changelog/185.doc @@ -0,0 +1 @@ +Improved explanations in report of what it is that Fil tracks, and what a flamegraph tells you. \ No newline at end of file From 691411b78c0fe938b0d0c70a9dac797f9cf6b222 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 13:04:51 -0500 Subject: [PATCH 3/7] News file. --- .changelog/292.misc | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 .changelog/292.misc diff --git a/.changelog/292.misc b/.changelog/292.misc new file mode 100644 index 00000000..e69de29b From 9bcb2dcb29adcb7e10c384d6ea788bb10e4c4953 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 13:05:01 -0500 Subject: [PATCH 4/7] Link to Sciagraph. --- filprofiler/_report.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/filprofiler/_report.py b/filprofiler/_report.py index 6bef2b98..83486f92 100644 --- a/filprofiler/_report.py +++ b/filprofiler/_report.py @@ -40,7 +40,7 @@ def render_report(output_path: str, now: datetime) -> str: margin: 4rem auto; font-size: 18px; }} - blockquote {{ border: 1px; border-color: black; }} + blockquote {{ border-width: 1px; border-color: black; border-style: solid; padding: 1em; }} div {{ text-align: center; }} @@ -90,11 +90,9 @@ def render_report(output_path: str, now: datetime) -> str:
Want memory and performance profiling for your production batch jobs? -I'm working on a -always-on profiler called Fil4prod that is fast and robust enough to run in production; -send me an email to participate -in the alpha program.
+I've also created an +always-on profiler called Sciagraph that is fast and robust enough to run in production.

Now what?

From 58e513dd83567a51fd1ef61c3e6499199c1356a9 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 13:06:21 -0500 Subject: [PATCH 5/7] Fix typo. --- .changelog/291.doc | 1 + docs/src/fil/api.md | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) create mode 100644 .changelog/291.doc diff --git a/.changelog/291.doc b/.changelog/291.doc new file mode 100644 index 00000000..7943e841 --- /dev/null +++ b/.changelog/291.doc @@ -0,0 +1 @@ +Fix bad command name in the API documentation, thanks to @kdebrab. \ No newline at end of file diff --git a/docs/src/fil/api.md b/docs/src/fil/api.md index 7bd37751..c12b6a91 100644 --- a/docs/src/fil/api.md +++ b/docs/src/fil/api.md @@ -59,10 +59,10 @@ $ python yourscript.py --config=myconfig Now you would do: ```console -$ filprofiler python yourscript.py --config=myconfig +$ fil-profile python yourscript.py --config=myconfig ``` -Notice that you're doing `filprofiler `**`python`**, rather than `filprofiler run` as you would if you were profiling the full script. +Notice that you're doing `fil-profile `**`python`**, rather than `fil-profile run` as you would if you were profiling the full script. Only functions running for the duration of the `filprofiler.api.profile()` call will have memory profiling enabled, including of course the function you pass in. The rest of the code will run at (close) to normal speed and configuration. From 5f888f878920afaa5a94e0177294e89983b993d4 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 13:23:48 -0500 Subject: [PATCH 6/7] Add button to open reports in a new window. --- .changelog/298.feature | 1 + filprofiler/_report.py | 20 +++++++++++--------- 2 files changed, 12 insertions(+), 9 deletions(-) create mode 100644 .changelog/298.feature diff --git a/.changelog/298.feature b/.changelog/298.feature new file mode 100644 index 00000000..894c32cf --- /dev/null +++ b/.changelog/298.feature @@ -0,0 +1 @@ +Reports now have a "open in new tab" button. Thanks to @petergaultney for the suggestion. \ No newline at end of file diff --git a/filprofiler/_report.py b/filprofiler/_report.py index 83486f92..53672c33 100644 --- a/filprofiler/_report.py +++ b/filprofiler/_report.py @@ -74,20 +74,22 @@ def render_report(output_path: str, now: datetime) -> str:

{argv}

Profiling result

-

-

- + +
+
+
+
+
Need help, or does something look wrong? Read the documentation, and if that doesn't help please file an issue and I'll try to help.
-
- -

-

-
Want memory and performance profiling for your production batch jobs? I've also created an @@ -95,7 +97,7 @@ def render_report(output_path: str, now: datetime) -> str: >always-on profiler called Sciagraph that is fast and robust enough to run in production.

-

Now what?

+

Learn how to reduce memory usage

Need help reducing your data processing application's memory use? Check out tips and tricks here.

From 82f01eb30129a8a9fa36d8d5e9706ee5badcb6b5 Mon Sep 17 00:00:00 2001 From: Itamar Turner-Trauring Date: Wed, 26 Jan 2022 13:49:58 -0500 Subject: [PATCH 7/7] Fix lint. --- filprofiler/_report.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/filprofiler/_report.py b/filprofiler/_report.py index 53672c33..83694246 100644 --- a/filprofiler/_report.py +++ b/filprofiler/_report.py @@ -126,7 +126,10 @@ def render_report(output_path: str, now: datetime) -> str:
  • If the data gets dumped from RAM to swap, Fil still counts it but it's not counted as resident in RAM.
  • If the memory is a large chunk of all zeros, on Linux no RAM is used by OS until you actually modify that memory, but Fil will still count it.
    • -
  • If you have memory that only gets freed on garbage collection (this will happen if you have circular references in your data structures), memory can be freed at inconsistent times across different runs, especially if you're using threads.
    • +
  • If you have memory that only gets freed on garbage collection +(this will happen if you have circular references in your data structures), +memory can be freed at inconsistent times across different runs, especially +if you're using threads.

See this article for more details.