Dependency mirroring RFC

[bvalyou]

I have dependency mirroring and mirror replication as requirements for my intended use case. I have some ideas for how to hack around it short term but would like to understand how mirroring (referenced in #153) is intended to work long term so I can limit rework later.

My guess is that there will be an argument to pkl project resolve like --mirror pkg.pkl-lang.org=example.com/pkl (supporting multiple mirrors) which would act as a replacer pattern for package manifests when pulling into the cache.

My current thought for working around it is to mirror the raw dependency (in an OCI repo, #354) and for each dependency of my project

• download the dependency explicitly and unpack the zip
• run a string replace based on the mirror, e.g. pkg.pkl-lang.org -> example.com/pkl from above, in the manifest (and assume for right now that all dependencies are in the manifest rather than inlined in an import in source)
• generate a PklProject file for the dependency based on the updated manifest and a defined folder structure
• recurse through its dependencies

Then use all resolved dependencies as local dependencies.

I'll also have to mirror all transitive dependencies on the inbound side but that's workable.

It would be a big help to understand if this is wildly different from the planned mirror support or if I'm over-engineering it (which wouldn't shock me).

Thanks!

[bioball]

I know you already said mirroring is a requirement, but: would HTTP proxying be good enough for you?

To support mirroring, we are considering rewrite rules. This is hand-wavey, but, possibly would work by adding something to the Pkl settings file. Something like:

groovy

// ~/.pkl/settings.pkl

packageRewriteRules {
  [#"https://pkg.pkl-lang.org/.*/(\w+)#] = "https://my.company.com/$1/$1"
}

But, we wouldn't implement this if HTTP proxying is good enough.

[bvalyou]

HTTP proxy is not good enough, unfortunately