Staging list --supersede -> results in weird issues

[DimStar77]

I somewhat think what I observe could be a regression from
#8946 probably together with a racing condition between when an SR is incoming and the execution order / runtime of the staging bot

'd observed it at least twice now - latest case this morning.

gambas3 (sr#766204) was staged in adi:11; a newer submission was incoming for the same package (sr#766549 at 10:20).

at 10:23, the package old package was decline by sbot (list --supersede). Which should then replace the package in adi:11 with the new one.

> osc rq show 766204
Request: #766204

  submit:       Education/gambas3@97 -> openSUSE:Factory


Message:
<no message>

State:   declined   2020-01-23T10:23:51 staging-bot
Comment: sr#766549 has newer source and is from the same project

Looks all right, but at the same time tbe bot crashed with:

Server returned an error: HTTP Error 403: Forbidden
No permission to decline request 766204
[script-executor] Script completed with exit code: 1.

The result now is that adi:11 still has 766204/gambas 3 tracked (state declined), but the package container in adi:11 no longer exists (package was unstaged according to the history)

Since 766204 is still tracked though, without a package being present, it is not possible to unstage it anymore - and a future run of staging list --supersede results in

Server returned an error: HTTP Error 404: Not Found
Error getting meta for project 'openSUSE:Factory:Staging:adi:11' package 'gambas3'

[DimStar77]

The easiest workaround to recover is to delete adi:11 completely and have a new adi be re-setup. I did not do that yet, in case somebody wants to look into more metadata of the proj which would vanish when I delete it

[coolo]

sbot killed the by_project review before trying to unstaging it:

• "195.135.221.2" staging-bot [23/Jan/2020:10:23:51 +0000] "POST /request/766204?cmd=changereviewstate&newstate=declined&by_project=openSUSE%3AFactory%3AStaging%3Aadi%3A11 HTTP/1.1" 200 53 "-" "osc/0.167.0" 0
• "195.135.221.2" staging-bot [23/Jan/2020:10:23:52 +0000] "DELETE /staging/openSUSE:Factory/staging_projects/openSUSE:Factory:Staging:adi:11/staged_requests HTTP/1.1" 403 114 "-" "osc/0.167.0" 2

That case noone thought of I'd say.

[Vogtinator]

Ping - this issue causes staging bot to grind to a halt quite frequently.

[Vogtinator]

Just worked around this for sr 781222 by reopening it, unselecting and declining again.

[coolo]

The workaround is merged.

[lkocman]

Well the requests auto-declined by bots are still a challenge. I have to use something like this construct otherwise I'll not hit the < 5 sec window of auto-decline done by licensedigger. This is quite painful and annoying tbh.

lkocman@localhost:~> osc -A https://api.opensuse.org request reopen 936691 -m "reopen to unstage" && osc -A https://api.opensuse.org staging -p openSUSE:Backports:SLE-15-SP4 unselect -m "declined" 936691 Result of change request state: ok

Normal worfklow would result into failure

lkocman@localhost:~> osc -A https://api.opensuse.org request reopen 936691 -m "reopen to unstage"
Result of change request state: ok
lkocman@localhost:~> osc -A https://api.opensuse.org staging -p openSUSE:Backports:SLE-15-SP4 unselect -m "declined" 936691
Unselecting "936691" from "openSUSE:Backports:SLE-15-SP4:Staging:adi:20"
Server returned an error: HTTP Error 403: Forbidden

[hennevogel]

what @lkocman meant was: I am not using the factory plugin so I don't benefit from the workaround 😃

[DimStar77]

lkocman@localhost:~> osc -A https://api.opensuse.org staging -p openSUSE:Backports:SLE-15-SP4 unselect -m "declined" 936691
Unselecting "936691" from "openSUSE:Backports:SLE-15-SP4:Staging:adi:20"

That DOES look like using the staging plugin (otoh, unselect in combination with -m implies 'unselect and add to ignore' - which is really not what you want to do when unselecting a declined request)

[DimStar77]

@lkocman >

lkocman@localhost:~> osc -A https://api.opensuse.org request reopen 936691 -m "reopen to unstage"
Result of change request state: ok

why do you reopen it? you can unselect a declined request (OBS internally reopens, unselects, declines)

[Vogtinator]

@lkocman >

lkocman@localhost:~> osc -A https://api.opensuse.org request reopen 936691 -m "reopen to unstage"
Result of change request state: ok

why do you reopen it? you can unselect a declined request (OBS internally reopens, unselects, declines)

That works neither for me nor @lkocman

[lkocman]

Correct, this is why I did the re-open

[DimStar77]

Ah - you are not target maintainers - that must be the difference there (Staging manager role seems to be not sufficient - which is indeed an issue then)