[Question] Metadata in Distributed Builds

[Minimonium]

The problem:
When performing multiple builds in parallel on different machines it's impossible to properly merge them out of the box on the upload machine.

Question:
Would it be possible to somehow help with that scenario?
Right now I naively merge the metadata files (recipe is guaranteed to be the same because of separate pre-build step to export it). Considering future improvements of metadata information it could break one day.

Would it be useful to have a command to regenerate the metadata? How would it work with revisions and such?

UPDATE:

I've ended up using the export-pkg command to merge build results from different workers on a single node.
I'm using the development flow to achieve that result, but there are some caveats to be aware of. So watch out.
Sadly the development flow is broken for a lot of recipes. Do not use it for the building of projects. Use conan create and manually store the package id somewhere to export-pkg the correct folder.

So we build using that and store the results in a packages/${CI_JOB_ID}/install for install files and packages/${CI_JOB_ID}/package for the resulted package. So far works great without any hacks!

[lasote]

Are you talking about the metadata.yml file, right?
There is no current mechanism to merge the informations, we implemented it as an internal file, nothing that should be manipulated by the users.
If I understand correctly, you are building packages in several "slaves" and then stashing the files to the master somehow so you can upload them all when the build is green?

[lasote]

You might be interested in this thread, but we really don't have a solution without using multiple repos: #871

[Minimonium]

I meant metadata.json, is yml a typo?

And yes, you've understood my situation right. I build on different environments on possibly different machines altogether. It's inconvenient to add an upload job for each build job instead of a generic single one.

While the issue seems interesting I have a different case. Before I want to upload a bunch - I need to somehow merge them, which is not possible natively now. Maybe through the export-pkg?

[lasote]

Yes, sorry it was a typo, I meant metadata.json.
Let me think about this.

[lasote]

I don't see a better alternative from what you are doing... the thing is, the revision is "calculated" during the export and the only place it is kept is the metadata... you cannot regenerate it safety from the files, because the revision could depend on the git commit of the recipe.

[Minimonium]

My concern is that the manual merge of metadata is not a supported feature and could break at any release (making the naive merge broken). I guess I need to explore the export-pkg.

[lasote]

Yes... I understand your concern, we shouldn't change that so easily but it could happen.

[ptc-ccrabb]

We're running into an identical situation. We need to run many builds on many different agents. We want all the builds to succeed, then to collect the packages from all the agents onto a single publishing agent. Then do a single publish job up to artifactory. It would be nice to get some support from Conan on this.

For example, a job kicks off to build MySpecialLib/1.0.0@owner/stream
On Agent 1 we build MacOS (x86_64 release and debug), iOS (x86_64 and armv8, both in release and debug)
On Agent 2 we build Linux (x86_64 release and debug)
On Agent 3 we build Windows (x86_64 Windows), WindowsStore (x86, x86_64, and arm64 WindowsStore)

On each agent I'd like to say conan export_package_to_fs MySpecialLib/1.0.0@owner/stream and it picks up all builds that agent did and provides a zip or tarball or some other file.

Then I'd be responsible for getting those to the publish agent where I'd like to say conan import_package_from_fs agent-1-exports.zip agent-2-exports.zip agent-3-exports.zip

In my usecase it'd be fine to assume that each zip contains builds for only one version of one package from one owner and one stream.

(edits fixed the syntax of the full package identifier)

[lasote]

Thanks for your feedback @ptc-ccrabb.
So, I'm going to summarize the current status of this. We think that the need you have of "sharing" the packages between slaves is a workaround because of lacking a CI-flow that works.
Our latest ideas (still not put in practice) are:

• Use package revisions
• Change the default package_id mode to recipe_revision_mode or package_revision_mode. That way, if A -> B -> C and C changes, the package_id for B and A will change (unless you specify further instructions in the recipes). The idea is to be sure that the packages are unique and never get mixed in a repository.
• Use lock files to coordinate the CI and build parallelize when possible: https://docs.conan.io/en/latest/versioning/lockfiles.html#how-to-use-lockfiles-in-ci
• Use a "tmp or stash or whatever" Conan repository where to upload the packages in the CI and then only when the build is green and everything is complete, promote the packages to a new repository ("develop", probably). For this promotion part, we still lack a piece of improving the "build-info" integration with Artifactory, but it is on my agenda.

Let me know what do you think.

The idea is to use the devops flows that are already working for other languages and technologies like Java Maven. So you can "promote" packages between repositories in order to follow the dev cycle, "develop", "QA", "release"...

[Minimonium]

I've updated the initial message with comments on the usage of export-pkg for anyone interested. It works great and solves the issue too! 🙂

[Minimonium]

@lasote An update on the situation. I rewrote everything to use a separate Conan repository like you suggested, with a conan copy on upload action to the main repository, since the build-info bit is missing for now.

Use a "tmp or stash or whatever" Conan repository where to upload the packages in the CI and then only when the build is green and everything is complete, promote the packages to a new repository ("develop", probably). For this promotion part, we still lack a piece of improving the "build-info" integration with Artifactory, but it is on my agenda.

The issue with export-pkg way is that it recalculates the package_id, which means that if there is some machine-specific logic in this function anywhere in the graph - for example Python detection in the Boost recipe, it'll generate an incorrect package_id.

[memsharded]

I think this ticket can be closed now.

Besides the above comments, there are new tools like cache save/restore of packages recently released (https://blog.conan.io/2023/11/28/Conan-new-features-2-0-14.html), or learned good practices with server side package promotions (supported by new tools like package-lists, and new work-in-progress commands in conan-extensions)