Snapshot rosdistro dependency metadata

[mini-1235]

Closes #144.

Summary

• store each package direct ROS dependencies in the existing, diffable rosdistro_snapshot.yaml
• derive source pins and dependency metadata from the same fetched rosdistro DistributionCache
• resolve recursive snapshot dependencies entirely from the local dependency lists
• keep the existing package.xml handling for recipe metadata, including the matching-live-cache optimization
• require snapshots without dependency metadata to be regenerated

A generated entry now looks like:

moveit_py:
  dependencies:
    - geometry_msgs
    - moveit_core
    - rclcpp
  tag: release/rolling/moveit_py/2.15.1-1
  url: https://github.com/ros2-gbp/moveit2-release.git
  version: 2.15.1

This avoids committing the complete compressed rosdistro cache or introducing a second snapshot file. Newly generated snapshots contain the metadata Vinca needs for recursive ROS dependency selection while remaining readable and reviewable.

Rolling results

• previous source-only snapshot: 231,904 bytes
• snapshot with direct dependencies: 518,567 bytes (about 506 KiB)
• complete uncompressed DistributionCache for comparison: 2,080,764 bytes
• snapshot generation: 6.10 seconds
• recipe generation: 63.54 seconds
• generated the expected two sbg_driver recipes

Validation

• pixi run fmt-check
• pixi run lint-check
• pixi run test (161 passed)
• git diff --check

[mini-1235]

@traversaro, is there a repository where I could refresh the snapshot and test this change? I originally planned to try it on Jazzy, but I noticed that its full-rebuild PR was recently merged and the rebuild has not yet completed due to some failures.

[mini-1235]

Or perhaps I could pick up either the Lyrical or Rolling one? Either would be fine with me.

[traversaro]

@traversaro, is there a repository where I could refresh the snapshot and test this change? I originally planned to try it on Jazzy, but I noticed that its full-rebuild PR was recently merged and the rebuild has not yet completed due to some failures.

I would try in rolling.

[traversaro]

One thing that I am a bit afraid is that the full cache (like https://repo.ros2.org/rosdistro_cache/rolling-cache.yaml.gz) is ~2 MB uncompressed, so I am not sure if we want to commit it to the repo. I wonder if we can store it somewhere else (but that is tricky to handle seamlessly) or we could somehow store only the dependencies information to avoid storing too much data.

[traversaro]

xref: ros/rosdistro#50112, I guess in bazel's ros central registry they had a similar problem.

[mini-1235]

One thing that I am a bit afraid is that the full cache (like https://repo.ros2.org/rosdistro_cache/rolling-cache.yaml.gz) is ~2 MB uncompressed, so I am not sure if we want to commit it to the repo. I wonder if we can store it somewhere else (but that is tricky to handle seamlessly) or we could somehow store only the dependencies information to avoid storing too much data.

I think we could keep the compressed version in the repository, no? The current Rolling cache is only about 211 KiB compressed

[traversaro]

One thing that I am a bit afraid is that the full cache (like https://repo.ros2.org/rosdistro_cache/rolling-cache.yaml.gz) is ~2 MB uncompressed, so I am not sure if we want to commit it to the repo. I wonder if we can store it somewhere else (but that is tricky to handle seamlessly) or we could somehow store only the dependencies information to avoid storing too much data.

I think we could keep the compressed version in the repository, no? The current Rolling cache is only about 211 KiB compressed

I do not know, I am not a big fan of adding compressed files to a git repo, as they remain in the history forever. But if that the easiest option, no objection for me, but I would like to hear the opinion also of @Tobias-Fischer, @ruben-arts or @baszalmstra .

[ruben-arts]

I might be misunderstanding the question, but if we can download the snapshot cache from the rosdistro repo releases then we can just do that right?

[baszalmstra]

Yeah I would prefer the solution from ros/rosdistro#50112 as well! I am also not a fan of storing the gzipped file in the history.

Alternatively we could unpack the files into directories (as text files) and let git do its regular magic.

[mini-1235]

I might be misunderstanding the question, but if we can download the snapshot cache from the rosdistro repo releases then we can just do that right?

To clarify, this PR currently does the following:

1. Downloads the current rosdistro cache once.
2. Creates an upstream DistributionCache from those exact bytes.
3. Uses that cache to obtain every package’s URL, tag, and version for rosdistro_snapshot.yaml.
4. Saves those same cache bytes as rosdistro_snapshot_cache.yaml.gz.

This ensures that the snapshot YAML and cache match when they are generated. If an immutable cache corresponding to that exact rosdistro state is available as a release asset, then downloading it later should work.

The potential mismatch only occurs if we later download the latest mutable cache: the snapshot YAML may still reference an older package version while the latest cache contains metadata for a newer version.

[traversaro]

Alternatively we could unpack the files into directories (as text files) and let git do its regular magic.

That would gives us ~2 MB for snapshot, not sure if I prefer that.

[baszalmstra]

Uses that cache to obtain every package’s URL, tag, and version for rosdistro_snapshot.yaml.

So between different runs, this information must match? Wouldn't it make more sense to store only that in a file then? Preferably a diff-able text file?

[traversaro]

I might be misunderstanding the question, but if we can download the snapshot cache from the rosdistro repo releases then we can just do that right?

No, we are downloading the cache of the package.xml of each repo from http://repo.ros2.org/rosdistro_cache, that only contains the latest version.

[mini-1235]

Uses that cache to obtain every package’s URL, tag, and version for rosdistro_snapshot.yaml.

So between different runs, this information must match? Wouldn't it make more sense to store only that in a file then? Preferably a diff-able text file?

If I understand correctly, this would be similar to storing the cache uncompressed. As @traversaro mentioned earlier, the downside is that it would add a roughly 2 MB file unless we post-process the cache and remove the data Vinca does not need.

[mini-1235]

post-process the cache and remove the data Vinca does not need

If this is the preferred direction, I can try updating this PR

[baszalmstra]

If we only extract the URL, tag, and version from the downloaded file, ideally we only cache that.

[traversaro]

If we only extract the URL, tag, and version from the downloaded file, ideally we only cache that.

That is what we already store in the snapshot file. But to quickly generate the recipes, we need at least also the dependencies

[baszalmstra]

Does it store it for the packages that are added, or for all packages in the distro?

[mini-1235]

Does it store it for the packages that are added, or for all packages in the distro?

All packages, something like this

[mini-1235]

I asked my agent to uncompress it and keep only the important information, so it now looks like this:

It is around ~500 KB now. What do you guys think about this approach?

[baszalmstra]

Thats still quite extensive but better than the gzipped content for sure! I would still prefer ros/rosdistro#50112 but while we wait this is the best solution I think!

[mini-1235]

Thats still quite extensive but better than the gzipped content for sure! I would still prefer ros/rosdistro#50112 but while we wait this is the best solution I think!

Great. If @traversaro also agrees with this approach, I can include it in my Lyrical full rebuild PR to make sure it doesn't break anything.

[traversaro]

If @traversaro also agrees with this approach

Go for me!

[traversaro]

Good to go for me!

[traversaro]

Sorry, I guess there was a conflict with #151, can you please fix the conflicts? Once that is done, it would be great if @baszalmstra could also put the thumbs up.

[mini-1235]

Thanks!