Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add report_invalidations to tabulate invalidations summary #303

Merged
merged 5 commits into from Jan 13, 2023

Conversation

charleskawczynski
Copy link
Contributor

@charleskawczynski charleskawczynski commented Sep 30, 2022

Hello πŸ‘‹πŸ»! There has been a lot of activity around hunting down invalidations lately, for example:

As mentioned in this comment of the discourse thread, I've added a function called report_invalidations to ReportMetrics.jl which summarizes the output of SnoopCompileCore.@snoopr with quantitative and actionable information.

I'd like to call this function from the julia-invalidations github action. I thought it might be cleaner to add this functionality to SnoopCompile directly (rather than a separate package) since it's relatively light weight (barring PrettyTables) and uses SnoopCompile internals.

The ReportMetrics.jl version of the printed table looks like this:

[ Info: Number of invalidations for SomeWork: 7
[ Info: SomeWork: 6 total method invalidations
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ <file name>:<line number>         β”‚ Method Name β”‚ Invalidations β”‚ Invalidations % β”‚
β”‚                                   β”‚             β”‚    Number     β”‚     (xα΅’/βˆ‘x)     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ SnoopCompile.jlirrationals.jl:182 β”‚  BigFloat   β”‚       1       β”‚       50        β”‚
β”‚ SnoopCompile.jlirrationals.jl:182 β”‚  BigFloat   β”‚       1       β”‚       50        β”‚
β”‚ SnoopCompile.jlirrationals.jl:192 β”‚   Float32   β”‚       0       β”‚        0        β”‚
β”‚ SnoopCompile.jlirrationals.jl:191 β”‚   Float64   β”‚       0       β”‚        0        β”‚
β”‚ SnoopCompile.jlirrationals.jl:192 β”‚   Float32   β”‚       0       β”‚        0        β”‚
β”‚ SnoopCompile.jlirrationals.jl:191 β”‚   Float64   β”‚       0       β”‚        0        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

@codecov
Copy link

codecov bot commented Sep 30, 2022

Codecov Report

Base: 84.08% // Head: 83.70% // Decreases project coverage by -0.38% ⚠️

Coverage data is based on head (5b4a42f) compared to base (dcae62d).
Patch coverage: 87.50% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #303      +/-   ##
==========================================
- Coverage   84.08%   83.70%   -0.39%     
==========================================
  Files          17       18       +1     
  Lines        2162     2221      +59     
==========================================
+ Hits         1818     1859      +41     
- Misses        344      362      +18     
Impacted Files Coverage Ξ”
src/invalidations.jl 74.06% <85.71%> (-3.74%) ⬇️
src/report_invalidations.jl 86.66% <86.66%> (ΓΈ)
src/SnoopCompile.jl 100.00% <100.00%> (ΓΈ)
src/parcel_snoopi_deep.jl 90.01% <0.00%> (+0.10%) ⬆️
src/parcel_snoopi.jl 95.00% <0.00%> (+0.85%) ⬆️
SnoopPrecompile/src/SnoopPrecompile.jl 94.11% <0.00%> (+6.61%) ⬆️
SnoopCompileCore/src/snoopi_deep.jl 100.00% <0.00%> (+8.00%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

β˜” View full report at Codecov.
πŸ“’ Do you have feedback about the report comment? Let us know in this issue.

@charleskawczynski
Copy link
Contributor Author

bump πŸ™‚

@charleskawczynski
Copy link
Contributor Author

Bump! πŸ™‚

@timholy
Copy link
Owner

timholy commented Dec 1, 2022

I think we could have this, but I'd rather it be loaded under a @require PrettyTables = ... block. Often the best practice is to define the function

"""
Lotsa docs

Using `tabulated_invalidations` requires that you first load the PrettyTables package.
"""
function tabulated_invalidations end      # this defines a function with no methods, but the docstring is available.

as a "stub" in the main package, and then in __init__()

function __init__()
    # other stuff
    @require PettyTables = sha include("tabulated.jl")
end

and put the tabulated_invalidations methods in "tabulated.jl".

@charleskawczynski
Copy link
Contributor Author

charleskawczynski commented Jan 1, 2023

Ok, this now works I think as expected, here's a mwe (assuming some packages are loadable):

import SnoopCompileCore
invalidations = SnoopCompileCore.@snoopr begin
   using ForwardDiff
end;
import SnoopCompile
using PrettyTables # needed for report_invalidations to be defined
SnoopCompile.report_invalidations(;
   invalidations,
   process_filename = x -> last(split(x, ".julia/packages/"))
)

Which produces

β”Œ Warning: Could not attribute the following delayed invalidations:
β”” @ SnoopCompile ~/Dropbox/Caltech/work/dev/clones/forks/SnoopCompile.jl/src/invalidations.jl:552
Any[MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:init,), _A} where _A<:Tuple{AbstractDict})), MethodInstance for eltype(::Type{T} where T<:Tuple{AbstractDict}), MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:maxlog, :form), _A} where _A<:Tuple{Int64, Type})), MethodInstance for eltype(::Type{T} where T<:Tuple{Int64, Type})] invalidated Core.MethodInstance[MethodInstance for StaticArrays.var"#s25#170"(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any), MethodInstance for StaticArrays._precompile_()]
Any[MethodInstance for eltype(::Type{T} where T<:Tuple{AbstractDict}), MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:init,), _A} where _A<:Tuple{AbstractDict}))] invalidated MethodInstance for MacroTools.allbindings(::Any, ::Vector{Any})
[ Info: 233 methods invalidated for 6 functions
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ <file name>:<line number>                   β”‚ Function Name β”‚ Invalidations β”‚ Invalidations % β”‚
β”‚                                             β”‚               β”‚               β”‚     (xα΅’/βˆ‘x)     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ ForwardDiff/QdStj/src/dual.jl:427           β”‚ promote_rule  β”‚      209      β”‚       82        β”‚
β”‚ StaticArrays/x7lS0/src/abstractarray.jl:150 β”‚    similar    β”‚      18       β”‚        7        β”‚
β”‚ StaticArrays/x7lS0/src/SizedArray.jl:88     β”‚    convert    β”‚      15       β”‚        6        β”‚
β”‚ StaticArrays/x7lS0/src/mapreduce.jl:253     β”‚   foldl##kw   β”‚       8       β”‚        3        β”‚
β”‚ ForwardDiff/QdStj/src/dual.jl:145           β”‚    isequal    β”‚       3       β”‚        1        β”‚
β”‚ StaticArrays/x7lS0/src/SOneTo.jl:57         β”‚  getproperty  β”‚       3       β”‚        1        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

@charleskawczynski
Copy link
Contributor Author

Based on JuliaDiff/ForwardDiff.jl#620 (comment) and JuliaDiff/ForwardDiff.jl#620 (comment), should we update how report_invalidations works? cc @ChrisRackauckas @timholy

@charleskawczynski charleskawczynski changed the title Add tabulated_invalidations to tabulate inv summary Add report_invalidations to tabulate invalidations summary Jan 10, 2023
@charleskawczynski charleskawczynski changed the title Add report_invalidations to tabulate invalidations summary Add report_invalidations to tabulate invalidations summary Jan 10, 2023
Copy link
Owner

@timholy timholy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks nearly merge-worthy. I don't think there are big changes needed in how you collect the invalidations, because:

  1. You suggest collecting them with SnoopCompileCore rather than SnoopCompile βœ”οΈ
  2. Nothing here uses a workload. Of course this summary will include many invalidations that may not affect the users' code. filtermod and precompile_blockers can restrict them to ones that affect particular modules or workloads. Are you wondering if you should support them?

src/invalidations.jl Outdated Show resolved Hide resolved
test/snoopr.jl Show resolved Hide resolved

Print a tabular summary of invalidations given:

- `job_name::String` the name of the job
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the role of the job_name? Does it have any functional significance? (Can I choose a name that's wrong or broken?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You know, I think we can get rid of it. It's not really doing anything. I added it in case I'd run several invalidation jobs, so that the tables could be differentiated, but 1) that can be printed before the call to report_invalidations and 2) I think they'd need to be separate sessions anyway. I'll remove it

- `job_name::String` the name of the job

An `@info` message also prints some information, for example
if the table has been truncated (upon request).
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be interpreted as saying "you have to request the @info message" whereas I think you mean "it was truncated by request," which in turn really means "it was truncated by specifying nrows."

But even this is confusing since nrows has a default value of 10, so you don't actually have to request it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good feedback. I'm thinking we can skip this comment and just do a better job documenting n_rows. I'll update to something that I think is clearer/simpler

src/invalidations.jl Show resolved Hide resolved
@charleskawczynski
Copy link
Contributor Author

I think one last thing that we could do, that would really put this over the top in terms of ease of use, would be if we followed through with CliMA/ReportMetrics.jl#7 (comment), and made the file / line number hypertext links through PrettyTables (which I think is supported here).

I think this can be a followup feature, though. Thumbs up to open an issue πŸ™‚

@timholy timholy merged commit 64a9a8e into timholy:master Jan 13, 2023
@timholy
Copy link
Owner

timholy commented Jan 13, 2023

Thanks! This is a nice addition. Release of SnoopCompile 2.10.0 is underway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants