Feedstocks not converted once in main

[bandersen23]

Bug:

I’ve had three recipes stuck in the main branch for 3-4 days and haven’t been built by CI yet.

They are griffe, mkdocstrings-python, and mkdocstrings-python-legacy.

It looks like a 403 error on Travis CI when I look at the GitHub actions log. @conda-forge/core - could you check this out?

[beckermr]

I deleted the conda-forge/mkdocstrings-python-legacy-feedstock repo to see if that helps.

[beckermr]

That didn't work. @chrisburr You've had success here before. Any ideas?

[chrisburr]

We ended up needing to open a ticket as it was a bug on their side. I'll make a reproducer and report it again.

[q-posev]

I suppose this issue is related to the following in the output log of the Create feedstocks workflow:

subprocess.CalledProcessError: Command '['conda', 'smithy', 'register-ci', '--without-appveyor', '--without-webservice', '--feedstock_directory', '/tmp/tmphbfrna8u__feedstocks/trexio-feedstock', '--organization', 'conda-forge']' returned non-zero exit status 1.

Does it prevent the recently added packages (e.g. trexio) from being pushed to the conda-forge feedstock?

[chrisburr]

Here is the previous issue #15891. I've opened a support ticket with Travis and will follow up here when I get a response.

Does it prevent the recently added packages (e.g. trexio) from being pushed to the conda-forge feedstock?

Yes, we can't push code to feedstocks before the various CI providers have been registered successfully.

[bandersen23]

Thanks for looking into this @chrisburr and @beckermr!

[bandersen23]

@chrisburr and @beckermr - I noticed all but mkdocstrings-python-legacy were built today - any thoughts on why that one isn't built yet?

[beckermr]

I'm not sure. Let's wait a few hours and see what happens. It's failed on both circle and Travis. I reset everything just now so maybe it will go through.

[bandersen23]

@beckermr Still no luck...

[jakirkham]

Possibly related to this ( conda-forge/status#126 )?

[beckermr]

Ahhhh. For the time period over the weekend you may be right! Sorry about that!

[bandersen23]

No worries! Any timeline for it getting migrated?

[beckermr]

We converted a recipe 36 minutes ago. So I am wondering if this is fully fixed and we just have to wait.

@chrisburr did you hear back from Travis CI help at all?

[chrisburr]

I chased them up this morning (~8 hours ago) but no response yet.

[jakirkham]

Given we are seeing some feedstocks convert and others not, wonder if it is worth deleting the partially converted feedstocks and then rerunning conversion. Just in case they have some funky state, which is causing us hang ups for whatever reason.

[beckermr]

I did that previously and yes we should try again!

[bandersen23]

Should I resubmit the recipe in a new PR?

[beckermr]

Nope. We'll keep working on it. So sorry for the delay.

[jakirkham]

Looks like a bunch of them just cleared. Though mkdocstrings-python-legacy is still in there. Maybe let's give it a bit and see if it also clears.

[beckermr]

Something's out of sync and the job to make the feedstocks is running twice. Hopefully the tokens are ok.

[jakirkham]

Agreed. The last job that ran didn't actually start until after the commit was pushed to remove the already converted recipes ( c0f2720 ). There was a job before that that canceled itself (seems there was one just recently like that as well).

[jakirkham]

Actually I don't think it is anything too weird. It took ~20mins to complete the conversion jobs and our cron jobs run every 10 minutes. That's just enough time to have 2 overlapping jobs like that. Probably less of an issue now that we have cleared a bunch of recipes out.

Though if we were looking to change things, we could run less frequently. Or somehow adjust the cron job to skip if one is already running. Though we are kind of already doing that in the job, but maybe there is a way GH Actions could handle that for us (alleviating the need for the job to even run).

[jakirkham]

It just ran on mkdocstrings-python-legacy and failed again. Only took ~2mins to run being the only recipe. Went ahead and just deleted that repo to start fresh. Maybe a 2nd fresh start will do better?

[jakirkham]

It seems to be stuck on CircleCI?

Edit: Deleted the repo again, which removed it from CircleCI. Retrying...

Edit 2: Same issue

[BastianZim]

Though if we were looking to change things, we could run less frequently. Or somehow adjust the cron job to skip if one is already running. Though we are kind of already doing that in the job, but maybe there is a way GH Actions could handle that for us (alleviating the need for the job to even run).

There is https://github.blog/changelog/2021-04-19-github-actions-limit-workflow-run-or-job-concurrency/ but last time I discussed it with @beckermr it seemed like it doesn't have all the features that we'd need.

[beckermr]

The code should already skip duplicate jobs. I wonder if that broke.

[jakirkham]

Not from what I could tell. It seemed to behave as expected. Please see this comment ( #18287 (comment) ). Should add the overlapping job did cancel itself.

[beckermr]

Ohhh great! Then I was confused. Yay!

[jakirkham]

No worries :)

[bandersen23]

Since every other feedstock seems to be created, it seems easiest to remove the recipe and to push it again... would that fix it? Would that fully reset the CI jobs?

[beckermr]

It would not. This is an issue in the circle and or Travis backends. We can make the feedstock without those services but we'd prefer not to.

[bandersen23]

I understand that. Any further thoughts on this?

[beckermr]

So I cannot even get this feedstock to build by hand in the circle ci app online.

[bandersen23]

That's odd... I just checked and was able to build it locally with boa so I don't think there's a problem with the recipe?

conda mambabuild mkdocstrings-python-legacy

[beckermr]

no I mean circleci running on the repo at all.

[jakirkham]

Yeah I had to create a config file in the repo to setup CircleCI

[beckermr]

I'm at a loss on what to do here. Maybe we remove circle for this one. Others are working just fine.

[jakirkham]

Is there an easy way to do that?

[beckermr]

We can edit the ci scripts on staged-recipes.

[isuruf]

Yes, let's disable circle-ci for now, convert mkdocstrings-python-legacy and re-enable circle-ci

[beckermr]

I'll do this tonight when staged-recipes activity has died down a bit.

[beckermr]

OK done!. We are so sorry for the delay with this one and thank you for waiting!

[jakirkham]

Thanks Matt! 😄

[bandersen23]

Thank you so much! No worries.