Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FR: add -z option to pgconvert (compress on the fly via zstd) #4

Open
GitMensch opened this issue Nov 15, 2022 · 2 comments
Open

FR: add -z option to pgconvert (compress on the fly via zstd) #4

GitMensch opened this issue Nov 15, 2022 · 2 comments

Comments

@GitMensch
Copy link
Contributor

similar to perf record:

       -z, --compression-level[=n]
           Produce compressed trace using specified level n (default: 1
           - fastest compression, 22 - smallest trace)

I think this would mean an (optional?) dependency on zlib and then "just" wrap the file output in a compression function using the level as specified / default and having the pgconvert tool using zlib to open the file.

The reason for this FR is that the CPU doing the profiling commonly has some spare cycles and reducing the data to write also leads to an io speedup (and long profiling always tends to fill the disk fast).

@ostash
Copy link
Owner

ostash commented Nov 15, 2022

If we want for pgcollect to affect the system minimally, I'd prefer zstd instead of zlib.
Did you encounter very large data files? I've used pgcollect for a HUGE application, running for quite a long period (more than 1 hour), with -F 1000 and the data file had a reasonable size.

Beware that pgcollect data files are much smaller than perf's ones.

@GitMensch
Copy link
Contributor Author

If we want for pgcollect to affect the system minimally, I'd prefer zstd instead of zlib.

Whatever you choose :-)

Did you encounter very large data files?

I've run with -F 8196 (I know "more data" is not necessarily "better data") to be able to compare the result in callgrind with perf + https://github.com/KDAB/hotspot (I think this was the default frequency used there) and then have seen bigger profiles with pgcollect than with perf record -z in first tests (the real app hangs on #2).

As noted: even minimal compression should be quite useful (of course that depends on the data).
Tested with the zstd command line utility, this results in quite a nice compression:

$> zstd -z outfile.pgdata
outfile.pgdata       :  3.50%   (16961792 => 593347 bytes, outfile.pgdata.zst)

@GitMensch GitMensch changed the title FR: add -z option to pgconvert (compress on the fly via zlib) FR: add -z option to pgconvert (compress on the fly via zstd) Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants