Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a benchmark for compilation times #1959

Merged
merged 15 commits into from
Mar 12, 2022
Merged

Add a benchmark for compilation times #1959

merged 15 commits into from
Mar 12, 2022

Conversation

rikhuijzer
Copy link
Collaborator

@rikhuijzer rikhuijzer commented Feb 28, 2022

Want meten is weten (because measuring is knowning). I've tried to do something fancy like rikhuijzer/PrecompileMacro.jl#1, but that will take some (a lot of) time to make. So, as a first step, this PR just adds a separate benchmark for compilation time. I think that I've missed a lot of entry points, but I hope that once we are able to pin down what spots are compiled exactly, then we can start making progress on improving the performance.

@github-actions
Copy link
Contributor

Try this Pull Request!

Open Julia and type:

julia> import Pkg
julia> Pkg.activate(temp=true)
julia> Pkg.add(url="https://github.com/rikhuijzer/Pluto.jl", rev="rh/precompile-macro")
julia> using Pluto

@rikhuijzer

This comment was marked as outdated.

@rikhuijzer
Copy link
Collaborator Author

I've added precompile directives to the other entrypoints inside SessionActions. I wasn't able to use PrecompileMacro.jl because it gave very weird errors. Adding the precompile directives show some improvements on the allocations in the logs:

5×3 DataFrame
 Row │ Operation                Allocations (MB)  Time (seconds)
     │ String                   Float64           Float64
─────┼───────────────────────────────────────────────────────────
   1 │ using Pluto                          76.5             0.8
   2 │ Pluto.ServerSession()                13.2             0.2
   3 │ SessionActions.open                 897.4            10.4
   4 │ SessionActions.shutdown              18.8             0.3
   5 │ SessionActions.new                   16.9             0.4

Don't take these times too seriously. This run was on a particularly slow runner. Look at allocations.

I guess this provides a good idea of the tradeoff between running time and adding these directives. In this case, the 3 extra directives save 0.4 seconds in total or so on my pc.

@rikhuijzer

This comment was marked as outdated.

@rikhuijzer

This comment was marked as outdated.

@fonsp
Copy link
Owner

fonsp commented Mar 5, 2022

I made JuliaPackaging/RelocatableFolders.jl#15 about this. Did you switch git branches?

@rikhuijzer

This comment was marked as outdated.

@rikhuijzer
Copy link
Collaborator Author

This is how the output looks now:

 ─────────────────────────────────────────────────────────────────────
                                           Time          Allocations
                                     ───────────────   ───────────────
           Total measured:                 740s            14.0GiB

 Section                     ncalls     time    %tot     alloc    %tot
 ─────────────────────────────────────────────────────────────────────
 import Pluto                     1    459ms    0.1%   66.0MiB    0.5%
 compiletimes.jl                  1    24.7s    3.4%   2.10GiB   15.3%
   SessionActions.open            1    12.5s    1.7%   1.14GiB    8.3%
   SessionActions.shutdown        1    563ms    0.1%   71.5MiB    0.5%
 Events.jl                        1    4.58s    0.6%    405MiB    2.9%
 WorkspaceManager.jl              1    93.3s   12.7%   1.01GiB    7.4%
 packages/Basic.jl                1     148s   20.1%    878MiB    6.2%
 Bonds.jl                         1    27.0s    3.7%    284MiB    2.0%
 RichOutput.jl                    1    45.1s    6.1%   1.31GiB    9.5%
 React.jl                         1     116s   15.8%   2.91GiB   21.2%
 Dynamic.jl                       1    31.5s    4.3%    799MiB    5.7%
 MacroAnalysis.jl                 1    98.9s   13.4%    825MiB    5.9%
 Logging.jl                       1    15.1s    2.0%   52.8MiB    0.4%
 webserver.jl                     1    20.6s    2.8%   1.52GiB   11.1%
 Notebook.jl                      1    87.5s   11.9%    568MiB    4.0%
 Configuration.jl                 1    8.27s    1.1%    402MiB    2.9%
 ReloadFromFile.jl                1    4.92s    0.7%    100MiB    0.7%
 packages/PkgCompat.jl            1    398ms    0.1%   33.1MiB    0.2%
 ExpressionExplorer.jl            1    5.28s    0.7%    317MiB    2.3%
 MethodSignatures.jl              1    132ms    0.0%   15.7MiB    0.1%
 MoreAnalysis.jl                  1    925ms    0.1%   79.9MiB    0.6%
 Analysis.jl                      1   94.6ms    0.0%   4.12MiB    0.0%
 webserver_utils.jl               1    818ms    0.1%   43.1MiB    0.3%
 data structures.jl               1    713ms    0.1%   68.3MiB    0.5%
 DependencyCache.jl               1    355ms    0.0%   27.4MiB    0.2%
 Throttled.jl                     1    841ms    0.1%   2.14MiB    0.0%
 cell_disabling.jl                1    536ms    0.1%   26.9MiB    0.2%
 ───────────────────────────────────────────────────────────────────── 

The compiletimes.jl runs first and gives an estimate of how long compilation takes.

@rikhuijzer
Copy link
Collaborator Author

@fonsp Can you review this?

@fonsp
Copy link
Owner

fonsp commented Mar 12, 2022

I took the sessionactions precompile statements out, let's do that in a new PR (also so that we can use these timing to see the difference!!)

@fonsp
Copy link
Owner

fonsp commented Mar 12, 2022

Thanks!! Looks great! Seeing allocations seems very useful in CI

@fonsp fonsp merged commit 78af954 into fonsp:main Mar 12, 2022
@fonsp
Copy link
Owner

fonsp commented Mar 12, 2022

precompile statements in #1977

@fonsp fonsp added the backend Concerning the julia server and runtime label Apr 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Concerning the julia server and runtime performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants