Don't package /etc/grid-security/certificates

[matyasselmeci]

There isn't much point to globus-gsi-sysconfig providing an empty /etc/grid-security/certificates directory, and putting it in the package causes problems for people that already have it, e.g. as a symlink to a shared filesystem.

[ellert]

You should be consistent and remove the same from the debian packaging.

This directory was added to the packaging not that long ago. It came from a change in the old GT project 26 Jul 2018, merged into GCT 11 Aug 2018. It was obviously added for a reason, so do we know if those reasons still apply?

[matyasselmeci]

This is the commit that introduced the change: globus/globus-toolkit@28d6baa
There's nothing in the description that says why, nor is it part of a PR; I'll ask Joe Bester if he remembers why he made that change.

[fscheiner]

@matyasselmeci

There isn't much point to globus-gsi-sysconfig providing an empty /etc/grid-security/certificates directory, and putting it in the package causes problems for people that already have it, e.g. as a symlink to a shared filesystem.

I tested this in a VM driven by CentOS 7: I precreated /etc/grid-security and symlinked /tmp/certificates as certificates there. No warnings during installation of globus-gsi-sysconfig. Also no warnings during removal, though the symlink and the /etc/grid-security dir were removed when I left the /etc/grid-security dir. But the actual /tmp/certificates directory is untouched afterwards.

Does this constitute a problem?

[brianhlin]

The issue seems to occur specifically when installing globus-gsi-sysconfig when /etc/grid-security/certificates points to a CVMFS directory:

ls -l /etc/grid-security/certificates
lrwxrwxrwx. 1 root root 49 Jun 27 08:35 /etc/grid-security/certificates -> /cvmfs/oasis.opensciencegrid.org/mis/certificates

Then yum install globus-gsi-sysconfig fails with the following:

  Installing : globus-gsi-sysconfig-9.2-1.el7.x86_64                                        1/1 
Error unpacking rpm package globus-gsi-sysconfig-9.2-1.el7.x86_64
error: unpacking of archive failed on file /etc/grid-security/certificates: cpio: chmod
error: globus-gsi-sysconfig-9.2-1.el7.x86_64: install failed

[fscheiner]

Then yum install globus-gsi-sysconfig fails with the following:

  Installing : globus-gsi-sysconfig-9.2-1.el7.x86_64                                        1/1 
Error unpacking rpm package globus-gsi-sysconfig-9.2-1.el7.x86_64
error: unpacking of archive failed on file /etc/grid-security/certificates: cpio: chmod
error: globus-gsi-sysconfig-9.2-1.el7.x86_64: install failed

That chmod "error" looks interesting: Maybe the chmod doesn't occur on the symlink but the actual directory the symlink points to. I actually have no experience with CVMFS, but according to the assumed corresponding GitHub project's README if it is a read-only FS and file permissions are read-only (i.e. immutable), too, that chmod will fail as it operates inside the FS and not on the mountpoint.

[brianhlin]

Yeah, I suspect that's the case and this failure would also occur for any read-only FS.

[fscheiner]

Yeah, I suspect that's the case and this failure would also occur for any read-only FS.

Most likely. But still, I wonder if it is the right way to try to adapt the RPM spec to environment specifics we can't know beforehand. I don't think it should be expected for a GNU/Linux distribution where packages are installed by copying their contents into place that /etc (or parts of it) is/are not writable.

[brianhlin]

Most likely. But still, I wonder if it is the right way to try to adapt the RPM spec to environment specifics we can't know beforehand. I don't think it should be expected for a GNU/Linux distribution where packages are installed by copying their contents into place that /etc (or parts of it) is/are not writable.

Yeah, I agree but at the same time, packages should only own directories or files that they actually need. Does globus-gsi-sysconfig actually need to own /etc/grid-security/certificates?

[msalle]

Most likely. But still, I wonder if it is the right way to try to adapt the RPM spec to environment specifics we can't know beforehand. I don't think it should be expected for a GNU/Linux distribution where packages are installed by copying their contents into place that /etc (or parts of it) is/are not writable.

Yeah, I agree but at the same time, packages should only own directories or files that they actually need. Does globus-gsi-sysconfig actually need to own /etc/grid-security/certificates?

It seems the code does at least require it to exist: It checks a set of directories, including /etc/grid-security/certificates, see globus_gsi_system_config.c#L1599 and globus_gsi_system_config.c#L379.

Requiring a directory to exist without owning it might be tricky. There is quite some guidance in https://docs.fedoraproject.org/en-US/packaging-guidelines/ see in particular File and Directory Ownership and File and Directory Dependencies, the latter stating: "Directory dependencies SHOULD ONLY be used to express the dependency on that directory existing, not on any other functionality of any other package that might provide that directory."

[matyasselmeci]

I think it requires a cert directory to exist but it could be one of several others:

• $X509_CERT_DIR
• $HOME/.globus/certificates
• $GLOBUS_LOCATION/share/certificates

How useful is a cert directory if it doesn't have any certificates in it?

[msalle]

I think it requires a cert directory to exist but it could be one of several others:

* $X509_CERT_DIR

* $HOME/.globus/certificates

* $GLOBUS_LOCATION/share/certificates

How useful is a cert directory if it doesn't have any certificates in it?

That is indeed a fair question. A reason could be to prevent looking in $GLOBUS_LOCATION/share/certificates which is used when the other doesn't exist, but that also doesn't seem to make sense.
Not sure what's best to do. It would still be good to know why Globus added it...

[matyasselmeci]

Unfortunately I got no response from Joe Bester on that subject.

[fscheiner]

Unfortunately I got no response from Joe Bester on that subject.

Maybe that change was introduced to be in sync with what is checked internally by globus-gsi-sysconfig (1st paragraph of #90 (comment)).

---

Other point:

Let us assume SSH host public and private keys were stored in /etc/ssh/keys instead of /etc/ssh. Usually the host keys are created on first start of the OpenSSH daemon, so that directory would be empty after the corresponding package would be installed. Now think about a "special" configuration (e.g. a cluster of front end machines) that would use parts of a r/o FS mounted at each node at /etc/ssh/keys, i.e. one dir per node/one set of keys per node, but same config for the OpenSSH daemon on all nodes. Now suddenly the installation process of the package fails because the respective dir cannot be chmodded to the desired FS permissions because the actual dir is r/o. Let's assume it didn't fail in the past because the package maintainers didn't set ownership for this directory in the past.

Now what? (1) Should the package maintainers remove the ownership of a default directory essential for operation of an OpenSSH deamon, or (2) should maybe the users of the OpenSSH daemon with the "special" configuration use configuration directives to reconfigure the location of the keys to the respective parts of their r/o dir?

[matyasselmeci]

Well, OpenSSH only needs to create the keys if they don't already exist (and it has been asked to create them -- there's an AUTOCREATE_SERVER_KEYS option in the sysconfig). (I think LIGO uses GSI-OpenSSH without server keys at all.)

To generalize, I think unless the software absolutely depends on that path and cannot be configured otherwise, it shouldn't fail to install if it cannot create that path. (Or at least a command-line flag to bypass it; yum/dnf don't have one but maybe rpm --excludepath would work.) This is especially true for /etc, which is supposed to be 'owned' by the local admin.

[fscheiner]

Well, OpenSSH only needs to create the keys if they don't already exist (and it has been asked to create them -- there's an AUTOCREATE_SERVER_KEYS option in the sysconfig). (I think LIGO uses GSI-OpenSSH without server keys at all.)

The problem is setting FS permissions with chmod on a directory that's on a r/o FS (see #90 (comment)). If in my example during package installation a chmod on the hypothetical path /etc/ssh/keys needs to happen it would fail the same way.

To generalize, I think unless the software absolutely depends on that path and cannot be configured otherwise, it shouldn't fail to install if it cannot create that path.

But for the SSH case for example you can also configure the daemon to use a configuration file or host keys from anywhere in the FS, so it actually doesn't absolutely depend on the default path /etc/ssh but it's still owned by the package with:

[...]
%files
%defattr(-,root,root)
[...]
%attr(0755,root,root) %dir %{_sysconfdir}/ssh
[...]

... (check the spec file in http://vault.centos.org/7.5.1804/os/Source/SPackages/openssh-7.4p1-16.el7.src.rpm for example). Hence I would consider that the baseline.

[msalle]

I understand the reasoning, and I kind of agree that we probably should not install /etc/grid-security/certificates. In any case the test is only running at runtime, not at install time, and at that point it makes no sense to check a directory if it would be empty.
I do wonder if we actually need /etc/grid-security itself, as is now in the commit?

[fscheiner]

@matyasselmeci @ellert @brianhlin @msalle
Let's continue the discussion but maybe change the approach:

What's the actual purpose of the globus-gsi-sysconfig package?

[ellert]

• I still would like to understand why the directory was added to the package, and whether those reasons still apply.
• As I stated earlier in the discussion: if the directory is removed from the packaging, it should be done consistently, i.e. both in the RPM spec and in the debian/* packaging. The current PR only removes it from the RPM spec.
• Regarding the question "should /etc/grid-security also be removed?" the answer is definitely no. Other packages install files in that directory and rely on this package for the directory ownership. If it is removed from here it must be added to some other package, which seems to be an unnecessary reshuffling.

[fscheiner]

• I still would like to understand why the directory was added to the package, and whether those reasons still apply.

Yes, it'd be good to know, why it was explicitly added.

• As I stated earlier in the discussion: if the directory is removed from the packaging, it should be done consistently, i.e. both in the RPM spec and in the debian/* packaging. The current PR only removes it from the RPM spec.

I updated the Debian packaging accordingly.

• Regarding the question "should /etc/grid-security also be removed?" the answer is definitely no. Other packages install files in that directory and rely on this package for the directory ownership. If it is removed from here it must be added to some other package, which seems to be an unnecessary reshuffling.

+1 from me

[fscheiner]

@bester
Can you please shed some light on why the /etc/grid/security/certificates dir was added to the globus-gsi-sysconfig package with globus/globus-toolkit@28d6baa? Thanks in advance.

[matyasselmeci]

The thing is, I don't think an empty /etc/grid-security/certificates directory is useful, and any RPM that installs certificates into that directory will necessarily create that directory as well. This addition hasn't done anything useful for OSG; instead, it has only caused problems.

[fscheiner]

The thing is, I don't think an empty /etc/grid-security/certificates directory is useful, and any RPM that installs certificates into that directory will necessarily create that directory as well.

The latter would also not work for your configuration, as for you /etc/grid-security/certificates points to a read-only location.

This addition hasn't done anything useful for OSG; instead, it has only caused problems.

I see your point, but still think it's a configuration issue. As an alternative approach, would it be feasible on your side to use a "link farm" instead, so not the /etc/grid-security/certificates dir points to the [...]/certificates dir in the CVMFS, but the files of each installed certificate in /etc/grid-security/certificates point to the respective files in the [...]/certificates dir in the CVMFS? I assume this could be created with cp -s [...], though this uses relative paths only and additional operations will be required to keep it updated.

See, I am a little afraid to merge this and make our new release, only to later find out, that we overlooked something that breaks things during installation for other sites or infrastructures than OSG. But maybe that's overly cautious.

[brianhlin]

As much as I'd like to see this merged, I vote that we punt on this instead of having it hold up the release.

[matyasselmeci]

Sure.

[fscheiner]

@matyasselmeci @brianhlin @ellert @msalle
Ok, changed my mind about the integration of this PR. @ellert found a few issues that would have been included in our proposed next release v6.2.20190829 and I'd therefore like to create a new candidate tag to include the fixes for them - so we don't knowingly make a buggy release. And as it's now anyhow a week later, I'd say, let's merge this PR, too and do some installation tests with the resulting packages and if that works without issues, ask PMC for a vote about this new tag.

What do you think?

@matyasselmeci @brianhlin
Or did you in the meantime find another solution for your problem which works with the source code as is?

[brianhlin]

That plan works for me. The site in question found a workaround for the issue but AFAICT, the package is in the wrong here.

[ellert]

I have no strong opinion either way. But there seems to be a majority in favour of the change.

[msalle]

Hi all, just to add I also have no strong opinion either way, so just go ahead.

[fscheiner]

Ok, I'll merge this now and create a new tag.

[fscheiner]

@matyasselmeci @brianhlin @ellert @msalle
Ok, I tested installation of "all" GCT related packages (with [...] libglobus* globus* myproxy*) on:

• CentOS 6 => works w/o problems
• CentOS 7 => works w/o problems
• OpenSUSE Leap 15.0 => "issues" with globus-simpleca

( 66/159) Installing: globus-simple-ca-5.1-lp150.1.1.noarch ....................................................................................................[done]
Additional rpm output:
 

    C e r t i f i c a t e    A u t h o r i t y    S e t u p

This script will setup a Certificate Authority for signing Globus
users certificates.  It will also generate a simple CA package
that can be distributed to the users of the CA.

The CA information about the certificates it distributes will
be kept in:

/var/lib/globus/simple_ca

The unique subject name for this CA is:

cn=Globus Simple CA, ou=simpleCA-opensuse-leap-15-x86-64, ou=GlobusTest, o=Grid

Installing new CA files to /etc/grid-security/certificates... done
Creating RPM source tarball... done
	globus_simple_ca_e2b2ab35.tar.gz

setting the default CA to: /O=Grid/OU=GlobusTest/OU=simpleCA-opensuse-leap-15-x86-64/CN=Globus Simple CA

linking /etc/grid-security/certificates/grid-security.conf.e2b2ab35 to
        /etc/grid-security/grid-security.conf

linking /etc/grid-security/certificates/globus-host-ssl.conf.e2b2ab35 to
        /etc/grid-security/globus-host-ssl.conf

linking /etc/grid-security/certificates/globus-user-ssl.conf.e2b2ab35 to
        /etc/grid-security/globus-user-ssl.conf


...done.

hostname: Name or service not known
hostname: Name or service not known

    /root/.globus/usercert_request.pem already exists
    /root/.globus/usercert.pem already exists
    /root/.globus/userkey.pem already exists

If you wish to overwrite, run the script again with -force.

ERROR: The file: /etc/grid-security/hostcert_request.pem is not readable

cp: cannot stat '/var/lib/globus/simple_ca/hostcert.pem': No such file or directory
/tmp/tmp.tk3HyLZLyb

It seems to be no problem for OpenSUSE Leap 15.0 (or zypper) as the installation of the packages exits with 0. And it seems like CentOS (or yum) omits this informational output from %post, as I didn't see anything similar there.

In an earlier try, I had already installed a GridFTP server in this OpenSUSE driven VM, so the /etc/grid-security/certificates dir was already existing. At that time I was asked to provide a PEM passphrase when the installation came to the globus-simpleca package.

So it looks like globus-simpleca places files in /etc/grid-security during installation, though it doesn't own that directory according to its RPM spec file and it also has no direct dependency on globus-gsi-sysconfig, so that change in this PR doesn't affect it anyhow. It looks like the placement of these files in /etc/grid-security/certificates is done by some separate script or binary - maybe grid-ca-create.

[ellert]

This error is not related to that the /etc/grid-security/certificates directory is no longer installed by the globus-gsi-sysconfig package. The error is a consequence of the earlier error:

hostname: Name or service not known
hostname: Name or service not known

The hostname command fails, and the globus-simple-ca %post scriptlet does not check for this.

grid-cert-request -cn `hostname -f` -host `hostname -f`

therefore is interpreted as

grid-cert-request -cn -host

i.e. it tries to create a USER certificate request for the user with common name "-host" and save the request as a user certificate request in /root/.globus/usercert_request.pem (i.e. root's $HOME), instead of

grid-cert-request -cn <hostname> -host <hostname>

which would request a HOST certificate and save the request in /etc/grid-security/hostcert_request.pem. The next step then fails because /etc/grid-security/hostcert_request.pem does not exist.

We could check for failing hostname command in the %post scriptlet I presume, i.e. instead of

if [ ! -f /etc/grid-security/hostcert.pem ] && \
   [ ! -f /etc/grid-security/hostcert_request.pem ] && \
   [ ! -f /etc/grid-security/hostkey.pem ]; then
    grid-cert-request -cn `hostname -f` -host `hostname -f`

we could do

hostname=`hostname -f 2>/dev/null`
if [ ! -f /etc/grid-security/hostcert.pem ] && \
   [ ! -f /etc/grid-security/hostcert_request.pem ] && \
   [ ! -f /etc/grid-security/hostkey.pem ] && \
   [ ! -z "$hostname" ]; then
    grid-cert-request -cn $hostname -host $hostname

We could also try to find out why the hostname command is failing.

[fscheiner]

This error is not related to that the /etc/grid-security/certificates directory is no longer installed by the globus-gsi-sysconfig package.

Yes, that's what I figured. My comment on that was more related to the fact, (1) that the globus-simpleca package places files in unowned dirs and (2) that I noticed a different behaviour after removing and reinstalling it, which lead me to the possibly wrong conclusion, that the issue was related to the /etc/grid-security/certificates dir in existence from an earlier setup of a GridFTP server in that VM.

The error is a consequence of the earlier error:

hostname: Name or service not known
hostname: Name or service not known

The hostname command fails, and the globus-simple-ca %post scriptlet does not check for this.

Ok, thanks for figuring that out. Last week I only confirmed that hostname worked after the installation tries, but didn't notice that hostname -f did not. The reason was a misconfiguration on my local network: The local hostname and the "singular" hostname as derived from DNS didn't match. After changing that, hostname -f is able to determine the FQDN and the installation succeeds as intended.

---

As the reason was a local misconfiguration, we can IMHO continue with the release of the GCT v6.2.20190906 and include the "fix" from PR #106 in the next release.