[EngAut] Automate Sync/Release the polakdot-sdk templates

[mordamax]

Origin: paritytech/polkadot-sdk#3155 (comment)

Who has requested: @kianenigma

OKR Q2 2024:
Objective 1: Improving the Polkadot developer experience
KR: New dev templates are shipped as independent repositories automatically upon each stable release with source of truth in polkadot-sdk

Problem:
paritytech/polkadot-sdk#3155
The existence of substrate-developer-hub in itself is questionable in my opinion. It is not well maintained, and I much rather everything to be in /paritytech, and ideally in polkadot-sdk repo.

Details
From @kianenigma
What we suggest doing is the following:

• Bring all templates of polkadot-sdk under one folder. /templates/solo-chain, /templates/parachain and /templates/minimal are the main 3. These are the source of truth and always kept up to date.
• Create repositories corresponding for each, called polkadot-sdk-solo-chain-template, polkadot-sdk-solo-parachain and polkadot-sdk-minimal-tempalte.
• Write a script that receives a commit hash somewhere in the stable branch as input, and mirrors the templates into their corresponding repo. This script needs to replace all Cargo.toml files to use the released crates.
• We run the above script upon each update of stable branch, i.e. every 3 month.

[rzadp]

Requested Github bot creation: https://github.com/paritytech/security/issues/45

[rzadp]

I have a first draft in the paritytech-stg organization.

The draft PR: paritytech-stg/polkadot-sdk#24
The current result (minimal template): https://github.com/paritytech-stg/polkadot-sdk-minimal-template

---

I had to remove a dependency on frame because it is not available as a published crate.
Used frame-* and sp-* dependencies instead, and re-declared some types that were only available in frame/src.

@kianenigma Could you take a look and let me know if the direction I took makes sense?

[kianenigma]

(cc-ing @gupnik as he might be faster to reply while I am attending to some urgent debugging)

I think now's the time to publish frame crate as well. Would be good to ask @EgorPopelyaev or @Morganamilo if it had any cargo-related issues? probably no reason, just because we didn't mark it.

[gupnik]

Thanks @rzadp for looking into this. It seems that I don't have access to https://github.com/paritytech-stg/polkadot-sdk-minimal-template. Could you add me please?

I had to remove a dependency on frame because it is not available as a published crate.

Yeah, I agree with @kianenigma to publish the frame crate instead of removing it altogether. Let me know if I can help in any way.

[rzadp]

It seems that I don't have access to https://github.com/paritytech-stg/polkadot-sdk-minimal-template. Could you add me please?

Ah right, made it public now @gupnik

[rzadp]

I have a problem building the current result template, I'm missing a ExtrinsicInclusionMode enum because it does not exists (yet) in the released sp_runtime crate.

@gupnik Are you able to help with that?

The real problem is that with the automation, I'm trying to use released crates from release-crates-io-v1.8.0 branch, using the template code from master. I cannot run the automation from the release branch yet, because the templates didn't exist in the monorepo back then.

---

Shall we wait until:

• The next crates release, which will include the templates code that works properly with the released code.
• frame crate starts being released.
• @kianenigma I see you also have some ongoing work changing the templates?

Or shall I keep working on it with what we have presently?

[gupnik]

@rzadp I think we need to get the frame crate released. Can we do that please?

[rzadp]

@EgorPopelyaev @Morganamilo Could we have the frame crate released please?

[kianenigma]

@kianenigma I see you also have some paritytech/polkadot-sdk#3597 changing the templates?

This is an experimental branch (as indicated by "DNM"), if finalized, it will be a new template, so feel free to work on the 3 that is already there.

[gupnik]

@rzadp This has been published: https://crates.io/crates/polkadot-sdk-frame. Are we good to proceed?

[rzadp]

@rzadp This has been published: https://crates.io/crates/polkadot-sdk-frame. Are we good to proceed?

Great, I will try it out.

[rzadp]

The crate is not a part of the most recent release branch release-polkadot-v1.9.0, so I will still have to do some additional dance around it.

[rzadp]

Sent a PR to psvm to include v1.9.0: paritytech/psvm#2

[rzadp]

I got stuck trying to use this newly released crate. I'm probably doing something stupid, @gupnik could you give me a hand please?

To reproduce my issue:

git clone https://github.com/paritytech-stg/polkadot-sdk-minimal-template.git
cd polkadot-sdk-minimal-template
cargo +nightly build

Issue:

error: Could not find `frame-system` in `dependencies` or `dev-dependencies` in `<snip>/pallets/template/Cargo.toml`!
  --> pallets/template/src/lib.rs:10:1
   |
10 | #[polkadot_sdk_frame::pallet]

I don't think I should be changing the template and adding individual frame-* to dependencies?

---

It compiles fine when building inside the workspace.
Inside the workspace, I'm using a modified import and changed references from frame to polkadot_sdk_frame:

- frame = { path = "../../../../substrate/frame", default-features = false, features = ["experimental","runtime",] }
+ polkadot-sdk-frame = { package = "frame", path = "../../../../substrate/frame", default-features = false, features = ["experimental", "runtime"], version = "0.1.0" }

It is because the package is still called frame in the workspace.

The diff of my changes in the workspace is here.

When yanked out of the workspace and using released crates, the aforementioned issue happens.

[gupnik]

@rzadp I will take a look. Thanks.

[gupnik]

Seems like this is caused by https://github.com/paritytech/polkadot-sdk/blob/ea5f4e9a4d78234eb68bbb9130a24b0598c7893a/substrate/frame/support/procedural/tools/src/lib.rs#L115 where we assume the crate's name to be frame. Will raise a PR with the fix soon.

CC: @kianenigma in case you have something to add.

[gupnik]

@rzadp paritytech/polkadot-sdk#3813 is merged now. Could we proceed here please?

[rzadp]

@rzadp paritytech/polkadot-sdk#3813 is merged now. Could we proceed here please?

Sure, I'll continue.

[rzadp]

I've checked and all seems good for me, however to really proceed I'm gonna have to wait until a new crate release happens, that includes this PR.

[gupnik]

I've checked and all seems good for me, however to really proceed I'm gonna have to wait until a new crate release happens, that includes this PR.

@rzadp A new version has been published. Please let me know if you need anything else.

[rzadp]

Hey @gupnik, I'm still running into exactly same issue as described before.

The reproduction is also the same - try to build the master of https://github.com/paritytech-stg/polkadot-sdk-minimal-template.

It has been updated to use the newly released crates, including polkadot-sdk-frame@0.1.1

The result is the same, no matter if I import it like this:

frame = { package = "polkadot-sdk-frame", version = "0.1.1", (...)

use frame;

Or like this:

polkadot-sdk-frame = { version = "0.1.1", (...)

use polkadot_sdk_frame;

[gupnik]

@rzadp Seems that one of the dependencies has not been updated yet: https://crates.io/crates/frame-support-procedural-tools. Have asked to get this fixed.

[gupnik]

@rzadp This is fixed now.

[rzadp]

@rzadp This is fixed now.

Looks good, my minimal template now successfully builds outside of the monorepo, and can run a local dev chain 👍

Next I'll check the parachain and solochain templates.

[rzadp]

The PR has been merged and tested out in polkadot-sdk.

Results:

• minimal template
• parachain template
• solochain template

---

Questions for next steps:

@kianenigma @gupnik Is there anything I can do further in order to improve those templates?

@EgorPopelyaev @Morganamilo What could be a trigger to turn it into a fully automatic job?
A push to a release-crates-io-v* branch? Note that the crates need to be already published when the job starts. Is there a tag that gets created when crate release finishes perhaps?

[PierreBesson]

In addition to updating the standalone repos consistently. It would be good that each of those repos have proper CI pipelines set up for Continuous building and releasing. ping @lovelaced

We (Devops Team) are very interested in having those templates continuously released as binary and docker image so we can directly deploy and try to onboard them on a testnet for example. This is the closest we can get of e2e testing of a community parachain. This is important for keeping good support of those templates in our deployment tooling (ansible, helm-chart) and ensure a good devops experience for parachain teams.

Even better would be if the release team (ping @EgorPopelyaev) can build those templates in advance of them being synced to their own repos so we avoid syncing broken templates to the master branch of paritytech/polkadot-sdk-*-template where parachain teams will use it as a base of their parachain node.

[rzadp]

build those templates in advance of them being synced to their own repos so we avoid syncing broken templates

@PierreBesson In case you missed it - The templates are being build before syncing, as well as after syncing.

[PierreBesson]

@rzadp cargo check && cargo test is nice to do but it's not a release. We are not building the final binary and eg. testing it with Zombienet to validate that the parachain onboards properly.

[rzadp]

I created this PR which we'll need to synchronize templates with recent version 1.10.0 and following ones.

@EgorPopelyaev Do you have a standard way of testing those templates inside the monorepo before a release, that could be re-used before syncing the templates? Could we collaborate on it?

[EgorPopelyaev]

@rzadp Unfortunately, no, we don't have such tests in place for now.

[rzadp]

Note: We agreed to kick off the synchronization job as part of the manual process upon the crate releases, for now.

[rzadp]

The last version of psvm has introduced two problems for this workflow - with aliased import of crates, and apparently with build dependencies.

The problem can be seen in this automated PR - crates frame/polkadot-sdk-frame and substrate-build-script-utils. I'll be looking into it next week.

[rzadp]

Note: the problem with frame/polkadot-sdk-frame seems to be caused by the fact that the crate is named frame on the 1.10.0 branch - but the (desired) name polkadot-sdk-frame is there on the 1.9.0 branch, and on master. I'm trying to figure out what happened.

[rzadp]

Another note:

The last version of psvm has introduced two problems for this workflow - with aliased import of crates, and apparently with build dependencies.

Correction - the problem is not only about build dependencies, but also regular dependencies. The newest version of psvm seems to not convert packages that previously worked - work example sc-telemetry. I'm trying to find a pattern which packages are not working, and why.

[rzadp]

I have pinpointed the issues I'm having to psvm preferring to use Plan.toml for determining versions instead of Cargo.lock (as before).

The functionality to read Cargo.lock is still there (as a fallback) and it continues to work for my case.

I have asked here.

[rzadp]

The issue with psvm has been overcome. (So my 4 previous comments above are resolved now).

And now we ran into another issue: paritytech/polkadot-sdk#4347

[gupnik]

The issue with psvm has been overcome. (So my 4 previous comments above are resolved now).

And now we ran into another issue: paritytech/polkadot-sdk#4347

@rzadp Now that this is resolved, can we push an update please?

[rzadp]

@gupnik Looks like we're almost there.

minimal and solochain templates have been just updated with version 1.11.0 and all seems well with those two.

However I've run into an issue with the parachain one (wonder when I'll stop running into issues with this task 😅)

It appears that parachain-template-runtime and pallet-parachain-template are published as crates, which conflicts with the usage of psvm tool which changes local dependencies to crate dependencies, but should not do it in this case.

I'm not sure why they're published - intentionally or some leftover?

I could add some workarounds to it - but first I'll try to figure out if those are needed on crates.io - if not, yanking them will solve my issue.

[gupnik]

minimal and solochain templates have been just updated with version 1.11.0 and all seems well with those two.

Thanks @rzadp. However, there's now a doc-only crate that's not available in the minimal template. Similarly, it's not using the README.md from the repo. It seems that this file is also not updated.

It appears that parachain-template-runtime and pallet-parachain-template are published as crates, which conflicts with the usage of psvm tool which changes local dependencies to crate dependencies, but should not do it in this case.

But I do see that it has publish=false. So, it should not have been published in the first place?

[rzadp]

But I do see that it has publish=false. So, it should not have been published in the first place?

I would think so.
There is nothing wrong with this crate existing in crates.io on itself, but in our case it messes up the template when using psvm.
On the other hand there are downloads of this crate so I don't know. @EgorPopelyaev @Morganamilo any idea?

---

Thanks @rzadp. However, there's now a doc-only crate that's not available in the minimal template. Similarly, it's not using the README.md from the repo. It seems that this file is also not updated.

Please look at release-crates-io-v1.11.0 branch, not master - that's the source of the update.

[rzadp]

However, there's now a doc-only crate

Ah, it'll probably cause issues when syncing the next release 🥲

[gupnik]

Please look at release-crates-io-v1.11.0 branch, not master - that's the source of the update.

I actually see only master.

[gupnik]

However, there's now a doc-only crate

Ah, it'll probably cause issues when syncing the next release 🥲

Can we push that as well please? We are planning to use these templates for PBA and would be good to have the doc-only crate in there as well.

[rzadp]

@gupnik I meant the release-crates-io-v1.11.0 branch in polkadot-sdk - it's been branched out of master so does not contain all the recent changes from master. For example readme is still empty at that branch.

[rzadp]

Can we push that as well please? We are planning to use these templates for PBA and would be good to have the doc-only crate in there as well.

I'm not sure. In the other thread there is explicit discussion to remove it: paritytech/polkadot-sdk#3155 (comment)
Also relevant piece: https://github.com/paritytech/polkadot-sdk/blob/73c89d308fefcedfc3619f0273e13b6623766b81/docs/sdk/Cargo.toml#L9-L10

[gupnik]

@gupnik I meant the release-crates-io-v1.11.0 branch in polkadot-sdk

Oh, got it. Thanks!

Can we push that as well please? We are planning to use these templates for PBA and would be good to have the doc-only crate in there as well.

I'm not sure. In the other thread there is explicit discussion to remove it: paritytech/polkadot-sdk#3155 (comment) Also relevant piece: https://github.com/paritytech/polkadot-sdk/blob/73c89d308fefcedfc3619f0273e13b6623766b81/docs/sdk/Cargo.toml#L9-L10

@kianenigma What are your thoughts here?

[rzadp]

minimal and solochain templates have been just updated with version 1.11.0 and all seems well with those two.

Note: The issue with updating the parachain template should be resolved with a 1.12.0 release.

[rzadp]

Note: The 1.12.0 has not appeared yet so I'm waiting for that.

[kianenigma]

It appears that parachain-template-runtime and pallet-parachain-template are published as crates, which conflicts with the usage of psvm tool which changes local dependencies to crate dependencies, but should not do it in this case.

I'm not sure why they're published - intentionally or some leftover?

I don't see any reason to publish them, please make a PR to the base repo and mark them as unpunlish, thank you :)

[rzadp]

I don't see any reason to publish them, please make a PR to the base repo and mark them as unpunlish, thank you :)

@Morganamilo has yanked the crates, so this problem is already solved :) And in the repo they're already marked as unpublish.

[rzadp]

After a small fix in psvm, the synchronization has worked with no issues.

All templates got automatically updated to 1.12.0.

[rzadp]

Closing as completed - I'll still try to help with the source templates, but the automation of synchronization is done.