attempt to re-pull missing signatures from a remote

[dustymabe]

If I have an rpm-ostree system that is initially not using gpg verification and I later convert it to use them I can get into a weird situation. In Fedora 25 right now we aren't including gpg signatures in the ISO image we create. When I install from the ISO I have gpg verification turned off. I then later enable gpg verification but the system fails to even "get off the ground" for an rpm-ostree upgrade because there is no signature on the system for the current commit that is deployed.

A snippet of the kickstart file I used is below:

ostreesetup --osname="fedora-atomic" --remote="fedora-atomic" --url="file:////run/install/repo/content/repo" --ref="fedora-atomic/25/x86_64/docker-host" --nogpg
%post --erroronfail                                                                                                                                    
rm -f /etc/ostree/remotes.d/fedora-atomic.conf                                                                                                         
# For F25                                                                                                                                              
ostree remote add --set=gpg-verify=false fedora-atomic 'http://dl.fedoraproject.org/pub/fedora/linux/atomic/25/'                                       
%end

After boot, I then configured the remote to do gpg verification:

-bash-4.3# cat /etc/ostree/remotes.d/fedora-atomic.conf 
[remote "fedora-atomic"]
url=http://dl.fedoraproject.org/pub/fedora/linux/atomic/25/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-25-primary

Then I tried to do an upgrade (Note, no upgrade is available at this time):

-bash-4.3# rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
       Version: 25.59 (2017-02-14 21:01:28)
        Commit: 5a9de04163a60c5f2771b3d307601fa4e2234f12b662c97e28290b7bcbdb43f0
        OSName: fedora-atomic
  GPGSignature: (unsigned)
-bash-4.3# rpm-ostree upgrade 
Updating from: fedora-atomic:fedora-atomic/25/x86_64/docker-host
error: GPG verification enabled, but no signatures found (use gpg-verify=false in remote config to disable)

However if I manually pull down the signed file I can then upgrade fine:

-bash-4.3# cd /ostree/repo/objects/5a/
-bash-4.3# curl -O -L https://kojipkgs.fedoraproject.org/atomic/25/objects/5a/9de04163a60c5f2771b3d307601fa4e2234f12b662c97e28290b7bcbdb43f0.commitmeta
 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   569  100   569    0     0   1096      0 --:--:-- --:--:-- --:--:--  1096
-bash-4.3# rpm-ostree upgrade
Updating from: fedora-atomic:fedora-atomic/25/x86_64/docker-host
  Signature made Tue 14 Feb 2017 04:02:36 PM EST using RSA key ID 4089D8F2FDB19C98
  Good signature from "Fedora 25 Primary <fedora-25-primary@fedoraproject.org>"


Scanning metadata: 3195                                                                                                                                
No upgrade available.

[cgwalters]

This is going to be messy to fix. I think my vote here is to plow forward instead on doing the reworked content model in f25.

[dustymabe]

so, i have been able to hit this in other cases as well. I know it shouldn't happen but if somehow a commit gets into the repo without being signed and a client tries to pull it, it will get into this state. I know it shouldn't happen where a commit gets in unsigned, but I'm experiencing this right now with f25 as I'm finding commits that get in without getting signed and working with patrick to understand why. Either way it would be good for the client systems to be able to overcome temporary server side issues without us having to tell them how to "work around" the issue every time it happens.

Also, with this technology we won't always be the one hosting the ostree repo and it will be much more straightforward if the client user tells an admin gpg sig verification failed for aaabbbccc and an admin fixes the problem server side and then simply lets the user know the problem is resolved rather than having to tell the user how to workaround it every time.

[cgwalters]

This issue was originally about fixing upgrades for the ISO path for new installs. You're now talking about making it work better when transitioning existing installations. That's fine, and we probably should do that. But there are a number of things that need to be unwound to do that properly - the main one is that we need to figure out whether we even can upgrade the remote config automatically. Right now it isn't part of the tree (or a package) formally.

[dustymabe]

This issue was originally about fixing upgrades for the ISO path for new installs. You're now talking about making it work better when transitioning existing installations.

yes. sorry for the pivot. the original description of the issue was the only way I could reliably recreate the problem, but I had seen the issue in the past manifest itself from time to time when we were initially starting to sign trees. I just saw it again last night when trying to do rpm-ostree deploy even on the newer 2017.2-2 version of rpm-ostree (which I know had some fixes for gpg verification stuff) and I asked patrick to look into why those commits weren't getting signed.

But there are a number of things that need to be unwound to do that properly - the main one is that we need to figure out whether we even can upgrade the remote config automatically. Right now it isn't part of the tree (or a package) formally.

I'm not actually proposing that we start managing the "remote config" as part of the upgrade. While this might allow us for interesting abilities in the future (i.e. rebasing to a different remote ostree repo (25->26)), it wasn't what I was suggesting. I don't know how valid this is, but I was just suggesting that if we encounter a "local commit" that is missing signed commit metadata, then try to query the remote to see if it exists there. If not, then error.

Another thing we could probably do is if the commitmeta does exist and is invalid then try to refetch it (in case there was some kind of corruption during download or fs corruption later), but that is really just a nice-to-have.

WDYT?

[cgwalters]

Yeah, it might not be hard to make the upgrader not error out if it's missing signatures for the current commit.

[dustymabe]

Yeah, it might not be hard to make the upgrader not error out if it's missing signatures for the current commit.

I'd prefer not to limit it to the "current commit". What about having it "retry" the pull of the commitmeta for any local commit that is missing the commitmeta?

[cgwalters]

We actually do that today in pull, so I am thinking the bug is elsewhere. Regardless I think we can make it work to turn on GPG for existing installs that might be missing signed commits.

[dustymabe]

We actually do that today in pull,

really? that is interesting. I think I've noticed this when running "upgrades" and/or "deploys".

[puiterwijk]

I just hit the same issue (pulled once when there wasn't a commitmeta file, then commitmeta was put in place, but rpm-ostree still said there was no signature found).
When looking at the web server logs, the latter pulls did request the .commitmeta file, and returned it with a 200 status code, but rpm-ostree did not put it in the /ostree/repo directory.
After downloading the file manually and putting it there, rpm-ostree upgrade worked again.

[cgwalters]

This is likely an ostree bug - I bet what's happening is we're detecting nothing changed, and aborting the transaction.

[jlebon]

This should be fixed by ostreedev/ostree#873.