Artifacts: Decrease use of AbstractPlatform and Dict{AbstractPlatform} to reduce Pkg invalidations

[mkitti]

The Artifacts standard library currently makes use of Base.BinaryPlatforms.AbstractPlatform for several keywords.
This is problematic because these methods become susceptible to invalidation.
In particular, Pkg.BinaryPlatforms causes many invalidations of methods in the standard system image because of this.

To alleviate this, all methods receiving AbstractPlatform will now attempt to convert the AbstractPlatform keyword
arguments to Platform. Also, use of Dict{AbstractPlatform} is removed.

To assist with this conversion, a generic conversion of AbstractPlatform to Platform is added to Base.BinaryPlatforms.

See #54073 (comment) to see the reduction in invalidations.

[mkitti]

Before this pull request:

julia> using SnoopCompileCore

julia> invalidations = @snoopr begin
           using Pkg
       end

julia> using SnoopCompile

julia> invalidation_trees(invalidations)
3-element Vector{SnoopCompile.MethodInvalidations}:
 inserting ==(a::Union{Pkg.BinaryPlatforms.FreeBSD, Pkg.BinaryPlatforms.Linux, Pkg.BinaryPlatforms.MacOS, Pkg.BinaryPlatforms.Windows}, b::Base.BinaryPlatforms.AbstractPlatform) @ Pkg.BinaryPlatforms ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Pkg/src/BinaryPlatforms_compat.jl:89 invalidated:
   backedges: 1: superseding ==(x, y) @ Base Base.jl:208 with MethodInstance for ==(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.AbstractPlatform) (4 children)

 inserting tags(::Pkg.BinaryPlatforms.UnknownPlatform) @ Pkg.BinaryPlatforms ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Pkg/src/BinaryPlatforms_compat.jl:17 invalidated:
   mt_backedges:  1: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.triplet(::Base.BinaryPlatforms.AbstractPlatform) (0 children)
                  2: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.os_version(::Base.BinaryPlatforms.AbstractPlatform) (1 children)
                  3: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.libgfortran_version(::Base.BinaryPlatforms.AbstractPlatform) (1 children)
                  4: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.cxxstring_abi(::Base.BinaryPlatforms.AbstractPlatform) (1 children)
                  5: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.libstdcxx_version(::Base.BinaryPlatforms.AbstractPlatform) (1 children)
                  6: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.arch(::Base.BinaryPlatforms.AbstractPlatform) (2 children)
                  7: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.os(::Base.BinaryPlatforms.AbstractPlatform) (2 children)
                  8: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.libc(::Base.BinaryPlatforms.AbstractPlatform) (2 children)
                  9: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.call_abi(::Base.BinaryPlatforms.AbstractPlatform) (2 children)
                 10: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for Base.BinaryPlatforms.platforms_match(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.Platform) (13 children)
                 11: signature Tuple{typeof(Base.BinaryPlatforms.tags), Base.BinaryPlatforms.AbstractPlatform} triggered MethodInstance for (::Base.BinaryPlatforms.var"#match_loss#50")(::Base.BinaryPlatforms.AbstractPlatform, ::Base.BinaryPlatforms.Platform) (30 children)

 inserting print(io::Pkg.UnstableIO, arg::Union{SubString{String}, String}) @ Pkg ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.12/Pkg/src/Pkg.jl:48 invalidated:
   backedges: 1: superseding print(xs...) @ Base coreio.jl:3 with MethodInstance for print(::Any, ::String) (7 children)
              2: superseding print(io::IO, s::Union{SubString{String}, String}) @ Base strings/io.jl:250 with MethodInstance for print(::IO, ::String) (374 children)
   1 mt_cache

After this pull request:

julia> using SnoopCompileCore

julia> invalidations = @snoopr begin
           using Pkg
       end

julia> using SnoopCompile

julia> invalidation_trees(invalidations)
1-element Vector{SnoopCompile.MethodInvalidations}:
 inserting print(io::Pkg.UnstableIO, arg::Union{SubString{String}, String}) @ Pkg ~/src/Pkg.jl/src/Pkg.jl:48 invalidated:
   backedges: 1: superseding print(xs...) @ Base coreio.jl:3 with MethodInstance for print(::Any, ::String) (7 children)
              2: superseding print(io::IO, s::Union{SubString{String}, String}) @ Base strings/io.jl:250 with MethodInstance for print(::IO, ::String) (435 children)
   1 mt_cache

[KristofferC]

Presumably, the use of AbstractPlatform is to allow people to create their own platforms that adhere to some specified interface. But if these can trivially be converted to Platform what is the point of having the AbstractPlatform-type at all? Are there AbstractPlatforms actually created somewhere in the ecosystem?

[mkitti]

Are there AbstractPlatforms actually created somewhere in the ecosystem?

Yes. The main use of Base.BinaryPlatforms.AbstractPlatform is Pkg.BinaryPlatforms.

I have proposed normalizing UnknownPlatform there such that all AbstractPlatform implementations are just wrappers around Platform. Conversion there is just a matter of unwrapping these wrapper types.

JuliaLang/Pkg.jl#3873

After that Pkg.jl PR, we can add the following in Pkg.BinaryPlatforms:

Base.convert(::Type{Platform}, p::PlatformUnion} = p.p

I did some extended exploration of the ecosystem here:

JuliaLang/Pkg.jl#3736 (comment)

AppBundler.jl also consumes the Pkg.BinaryPlatforms API.
PeaceFounder/AppBundler.jl#4

@ararslan also has some uses in scripts, apparently.

As in this pull request, there can be a default conversion to Platform via the implicit interface in Base.BinaryPlatforms.

[giordano]

Are there AbstractPlatforms actually created somewhere in the ecosystem?

Yes: https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/b1d6c16dca0b0c6adab208ab1f7056777be7defc/src/Platforms.jl#L13

[mkitti]

The generic conversion in this pull request works for AnyPlatform, but showing the converted Platform fails.

julia> p = AnyPlatform()
AnyPlatform

julia> p2 = Platform(p);

julia> p2.tags
Dict{String, String} with 2 entries:
  "arch" => "any"
  "os"   => "any"

julia> p2.compare_strategies
Dict{String, Function}()

julia> Platform(p)
Error showing value of type Platform:
ERROR: KeyError: key "any" not found
Stacktrace:
  [1] getindex
    @ Base ./dict.jl:498 [inlined]
  [2] platform_name(p::Platform)
    @ Base.BinaryPlatforms ./binaryplatforms.jl:436
  [3] show(io::IOContext{Base.TTY}, p::Platform)
    @ Base.BinaryPlatforms ./binaryplatforms.jl:187
  [4] show(io::IOContext{Base.TTY}, ::MIME{Symbol("text/plain")}, x::Platform)

[mkitti]

Is this a problem?

julia> triplet(p2) # Platform
"any-unknown"

julia> triplet(p) # AnyPlatform
"any"

[giordano]

I don't think AnyPlatform makes any sense as Platform, that's the reason why it isn't an instance of Platform but its own subtype of AbstractPlatform. So please don't remove that.

[mkitti]

I don't think AnyPlatform makes any sense as Platform, that's the reason why it isn't an instance of Platform but its own subtype of AbstractPlatform. So please don't remove that.

Conceptually, I agree, but from a pragmatic standpoint it would help type stability if we could avoid use of Dict{AbstractPlatform}.

I think with some small modifications we could make a Platform that from the methods in Base.BinaryPlatforms that acts the same. For example, we could add a triplet tag.

[timholy]

I haven't looked into this at all (thanks for doing so, @mkitti), but in case you're not aware of it, I've often found it useful to define a getproperty method for abstract types to give them a real interface: https://timholy.github.io/SnoopCompile.jl/stable/snoopr/#Inferrable-field-access-for-abstract-types. The biggest success story for this trick was with the IO stack, which used to a huge magnet for invalidations (and sometimes is still tricky, but nowhere close to what it was once like).

[mkitti]

I'm starting to think about a PlatformWrapper that would act similarly to IOContext in terms of being a concrete type we could use.

struct PlatformWrapper{P} <: AbstractPlatform
    p::P
end

I need to think about this further if that would actually help. In some ways, the implementation of Platform already allows it to act a PlatformWrapper that just eagerly consumes the underlying AbstractPlatform. PlatformWrapper would just be a lazy version.

Note that we have also have an IO related invalidation issue that we are also working on with regard to Pkg.jl.

[mkitti]

The current invalidation issue originates with a Dict{AbstractPlatform} in Artifacts.jl leading to a bunch of potential dynamic dispatch that can be invalidated easily by just adding a new AbstractPlatform.

Maybe we need to do some inference magic here to force dynamic dispatch so that there is nothing to invalidate?

[timholy]

Yeah, sprinkling a few Base.@nospecializeinfer around seems likely to fix that nicely.

[mkitti]

Just so we are clear about the invalidation problem, here is the minimum working example.

julia> versioninfo()
Julia Version 1.11.0-beta1
Commit 08e1fc0abb9 (2024-04-10 08:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)


julia> using Base.BinaryPlatforms, SnoopCompileCore
       invalidations = @snoopr begin
           struct AnyPlatform <: AbstractPlatform end
           Base.BinaryPlatforms.tags(p::AnyPlatform) = Dict{String,String}()
       end
145-element Vector{Any}:
   MethodInstance for Base.BinaryPlatforms.platforms_match(::AbstractPlatform, ::Platform)
  0
   MethodInstance for (::Base.BinaryPlatforms.var"#47#49"{Platform})(::AbstractPlatform)
  1
   MethodInstance for Base.mapfilter(::Base.BinaryPlatforms.var"#47#49"{Platform}, ::typeof(push!), ::Base.KeySet{AbstractPlatform, Dict{AbstractPlatform, Dict{String, Any}}}, ::Set{AbstractPlatform})
  2
   MethodInstance for filter(::Base.BinaryPlatforms.var"#47#49"{Platform}, ::Base.KeySet{AbstractPlatform, Dict{AbstractPlatform, Dict{String, Any}}})
  3
...

[timholy]

Have you experimented with the following?

module BinaryPlatforms

Base.Experimental.@max_methods 1

# rest of module definition

end

[mkitti]

Base.Experimental.@max_methods 1

This does not quite work because there there is only one subtype of AbstractPlatform in Base.

julia> subtypes(Base.BinaryPlatforms.AbstractPlatform)
1-element Vector{Any}:
 Base.BinaryPlatforms.Platform

julia> Base.BinaryPlatforms.tags |> methods
# 1 method for generic function "tags" from Base.BinaryPlatforms:
 [1] tags(p::Base.BinaryPlatforms.Platform)
     @ binaryplatforms.jl:150

[mkitti]

But if these can trivially be converted to Platform what is the point of having the AbstractPlatform-type at all?

Another motivation I found is that some folks like to dispatch on on Windows or Linux:
PeaceFounder/AppBundler.jl#4 (comment)

[timholy]

Can you call tags from an invokelatest?

[mkitti]

Background

The definition of Platform is as follows.

struct Platform <: AbstractPlatform
    tags::Dict{String,String}
    # The "compare strategy" allows selective overriding on how a tag is compared
    compare_strategies::Dict{String,Function}
    # ...
end
tags(p::Platform) = p.tags

tags is just a Dict{String,String}. Most methods are implemented by just retrieving information from the Dict.

arch(p::AbstractPlatform) = get(tags(p), "arch", nothing)
os(p::AbstractPlatform) = get(tags(p), "os", nothing)
libc(p::AbstractPlatform) = get(tags(p), "libc", nothing)
call_abi(p::AbstractPlatform) = get(tags(p), "call_abi", nothing)
libgfortran_version(p::AbstractPlatform) = VNorNothing(tags(p), "libgfortran_version")
libstdcxx_version(p::AbstractPlatform) = VNorNothing(tags(p), "libstdcxx_version")
cxxstring_abi(p::AbstractPlatform) = get(tags(p), "cxxstring_abi", nothing)
os_version(p::AbstractPlatform) = VNorNothing(tags(p), "os_version")

Options

I see two options from here.

1. We call tags from invokelatest from the following 11 functions.
   • triplet
   • os_version
   • libgfortran_version
   • cxxstring_abi
   • libstdcxx_version
   • arch
   • os
   • libc
   • call_abi
   • platforms_match
   • match_loss
2. We continue on the path of converting everything to Platform and just store all the information in tags.

Forcing dynamic dispatch via invokelatest

I could keep exploring how to make this fully dynamic. It would make things slightly more extensible. However, it seems like an inferior solution if we can figure out how to do this statically. I think we are pretty close to the other solution.

Converting everything to Platform

("os", "arch", "libc", "call_abi", "libgfortran_version", "libstdcxx_version", "cxxstring_abi", "os_version") are already stored as tags.

The only other aspect that needs to be stored as a tag is triplet which is currently computed from the others as follows.

function triplet(p::AbstractPlatform)
    str = string(
        arch(p)::Union{Symbol,String},
        os_str(p), 
        libc_str(p),
        call_abi_str(p),
    )

    # Tack on optional compiler ABI flags
    if libgfortran_version(p) !== nothing
        str = string(str, "-libgfortran", libgfortran_version(p).major)
    end
    if cxxstring_abi(p) !== nothing
        str = string(str, "-", cxxstring_abi(p))
    end
    if libstdcxx_version(p) !== nothing
        str = string(str, "-libstdcxx", libstdcxx_version(p).patch)
    end

    # Tack on all extra tags
    for (tag, val) in tags(p)
        if tag ∈ ("os", "arch", "libc", "call_abi", "libgfortran_version", "libstdcxx_version", "cxxstring_abi", "os_version")
            continue
        end
        str = string(str, "-", tag, "+", val)
    end
    return str
end

It seems that the only case that needs special handling for triplet is AnyPlatform as mentioned by @giordano above. To handle that we could just create a special case for triplet to be any when os and arch are any. We could hard code that logic into the triplet(::AbstractPlatform) method above or create a new triplet tag to handle this.

[giordano]

I don't think AnyPlatform needs special treatment? I don't think Pkg ever deals directly with platforms defined elsewhere. AnyPlatform is only used internally in BinaryBuilder.

[mkitti]

We are not talking about Pkg here. This is about Base and Artifacts as a standard library built into the system image.

Pkg, like BinaryBuilderBase, introduces new AbstractPlatform subtypes causing invalidations due to the use of Dict{AbstractPlatform} in Artifacts.jl.

As you pointed out, BinaryBuilderBase.jl does call artifacts_meta in Artifacts, possibly passing in AnyPlatform() which this PR would convert to Platform("any", "any").

https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/b1d6c16dca0b0c6adab208ab1f7056777be7defc/src/Prefix.jl#L751

Showing Platform("any", "any") does throw an error though which is what I'm trying to handle here.

[giordano]

As you pointed out, BinaryBuilderBase.jl does call artifacts_meta in Artifacts, possibly passing in AnyPlatform() which this PR would convert to Platform("any", "any").

No, we pass a concrete target platform, which is a Platform object, never AnyPlatform. I still don't think AnyPlatform needs special treatment. My only point in #54073 (comment) was not to remove AbstractPlatform.

[mkitti]

I'm have minimized the diff. We do not need any method signature changes or additions. All we need is conversion to Platform.

[mkitti]

I'm minimizing the diff.

Also I've submitted to a similar pull request to Pkg.jl which makes it harder to invalidate methods there when loading a package (such as BinaryBuilderBase) that declares new methods on a new AbstractPlatform type.
JuliaLang/Pkg.jl#3874

I have also included here some a revision to get_platform so it does not error when showing a non-canonical Platform.

julia> using BinaryBuilderBase

julia> convert(Platform, AnyPlatform())
any any

[mkitti]

@JanisErdmanis and @ararslan , is this pull request compatbile with your usage?

[JanisErdmanis]

It would work for me. Perhaps, looking at the changes, I would prefer restricting the type signature of the methods to Platform rather than doing conversion as it would make the code more concise.

I see that all the methods in binaryplatforms.jl are tightly coupled to the Platform object. It does not look like AbstractPlatform is useful at this level. Personally, I would not have an issue with defining it in AppBundler.jl or taking it from Pkg where platforms are defined as:

struct Linux <: AbstractPlatform
    platform::Platform
end

If you could add a corresponding convert methods there, it would be pretty great for my use case.

[mkitti]

It would work for me. Perhaps, looking at the changes, I would prefer restricting the type signature of the methods to Platform rather than doing conversion as it would make the code more concise.

Sure, but that would be changing the interface of a standard library package. The current approach at least allows for some reasonable expectation of backwards compatibility.

The only case I have seen so far is AnyPlatform where triplet is not standard. I am thinking of how to address that possibility - such as checking triplet and adding it as a tag.

If you could add a corresponding convert methods there, it would be pretty great for my use case.

See JuliaLang/Pkg.jl#3874

Thank you for taking a look.