Dist removal

[mbargull]

Description

Work in progress to remove legacy conda.models.dist.Dist usage.
Excerpt from gh-5069 :

[...] conda-build's slow (and memory intensive) ramp up before it gets to its actual first environment solving.
[...] work on purging legacy behavior/code such that we could at least prepare to make use of the deferred creation of package records.
A first step in that direction would be to finally remove conda-build's usage of the old Dist objects (removed from conda's core workings in 2018).

---

Impact

• Prepare usage of newer/more performant package index handling in conda-build.
• Removal and possible removal tech debt not only in conda-build but also conda.
• Breaking changes for library uses of conda_build:
  • not only might we remove Dist-related functions which might be used downstream,
  • but also not using a Dist-based package index anymore could break downstream expectations.

Closes gh-5069

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[mbargull]

Pull request overiew:

Goal

• Change necessary parts to remove the usage of Dist.
• Do not change how environment creations happen just yet.
  => This means: Actual runtime benefits (time/memory reduction) should be done after the Dist removal/cleanup work. (Ideally in subsequent PRs.)

Strategy

• Do not change behavior.
• Work iteratively and surgically.
• Copy over conda.models.dist and conda.plan (+conda.instructions).
  (These still exist (nearly) completely for conda-build only and could otherwise be removed from conda.)
• Iteratively reduce the surface of Dist/plan interaction.
  • remove Dist usage and unneeded functionality from Dist and plan
    (e.g., conda-build only creates, never updates environments => only create/link parts of plan are actually used)

Progress

• Dist objects have been removed.
• Test suite not yet run (hence opening this draft PR).
  • (Expecting failures; only ran local conda-forge/zlib-feedstock builds not yet more of the test suite, yet.)
• Removed most unneeded/defunct code regarding plan.

To do

• More cleanup needed.
• Better document the changes (i.e., fix commit messages to be more descriptive)

[mbargull]

FYI I just opened #5151 to remove all 24.1.0 deprecations

Thanks, I'll merge this gh-5151 here once it's in.

[kenodegard]

looking good! 🚀

[mbargull]

Thanks everyone for reviewing!

[kenodegard]

@mbargull phew! thanks soooo much for working on this (and being patient with the slooooow CI)! we would not have gotten this done nearly as quickly if you hadn't helped out with the Dist removal

BTW since you have merge rights can you click auto-merge and update the squashed commit message?

[mbargull]

(and being patient with the slooooow CI)

All good :).
I was also surprised about the skeleton test failure and had to dig a bit to understand/remember those permission issues (previously encountered those some years ago with clean up code for go packages which marked directories as read-only).
At some point I'd also like to continue to look into the parts in conda-build that make it so slow on Windows/macOS so we may get sub 30min CI times on all platforms 🤞 :).

BTW since you have merge rights can you click auto-merge and update the squashed commit message?

Oh right, good idea! Done.