Retrieved "hostkey.pem" seems to be undecrypted when installing from source

[iWangJiaxiang]

Hi guys.

I cloned latest GCT source 4b1ec65 and installed GCT using a non-root user gct to path /usr/local/gct6.

Configuring non-standard installation GCT instance was hard and I got stuck when launching globus-gatekeeper, which was caused by undecrypted hostkey.pem.

The error was:

GSS failed getting server credentials:
GSS Major Status: General failure
GSS Minor Status Error Chain:
globus_gsi_gssapi: Error with GSI credential
globus_gsi_gssapi: Error with gss credential handle
globus_credential: Valid credentials could not be found in any of the possible locations specified by the credential search order.
Valid credentials could not be found in any of the possible locations specified by the credential search order.
Attempt 1
globus_credential: Error reading host credential
globus_credential: Key is password protected: GSI does not currently support password protected private keys.
OpenSSL Error: pem_pkey.c:117: in library: PEM routines, function PEM_READ_BIO_PRIVATEKEY: bad password read
Attempt 2
globus_credential: Error reading proxy credential
globus_sysconfig: Could not find a valid proxy certificate file location
globus_sysconfig: Error with key filename
globus_sysconfig: File does not exist: /tmp/x509up_u1002 is not a valid file
Attempt 3
globus_credential: Error reading user credential
globus_credential: Key is password protected: GSI does not currently support password protected private keys.
OpenSSL Error: pem_pkey.c:117: in library: PEM routines, function PEM_READ_BIO_PRIVATEKEY: bad password read
Failure: GSS failed to get server credentials

Failed to start globus-gatekeeper

After long time search, I found solution from mail list Re: [gt-user] Problem with globus-gatekeeper by using command openssl rsa -in /etc/grid-security/hostkey.pem -out /etc/grid-security/hostkey.pem to remove the password.

It worked and I managed to launch globus-gatekeeper. I don't know if there would be more problems but it was a temporary work-around.

I don't know what exactly happened but the myproxy-retrieve command might be the root cause.

Hope you could give me some hints :D

Thanks

[iWangJiaxiang]

Got it! I checked the source below.

    /* host credentials should not be encrypted */
    if (getuid() == 0) {
        SSL_CREDENTIALS *creds;

        creds = ssl_credentials_new();
        ssl_private_key_load_from_file(creds, keyfile,
                                       client_request->passphrase, NULL);
        ssl_private_key_store_to_file(creds, keyfile, NULL);
        ssl_credentials_destroy(creds);
    }

It seems the myproxy-retrieve command not decrypting key file by default unless the executing user is root, which means the retrieved key file was treated as hostkey.

---

The official doc suggests not to user root for installation from source.

However, the problems I faced seems to tell the GCT should be configured as root user.

What should I do?

[msalle]

Hi,
As the error message said: GSI does not currently support password protected private keys.
which comes from globus_gsi_credential.c line 1171. The code there actually would allow for a callback (pw_cb) that could retrieve the password from the user (this is done in for example grid-proxy-init), but this is not implemented for any of the GSI services.
This is actually quite a common pattern for services: they usually don't have a passphrase on their hostcredentials since you typically like to be able to restart the hosts without sysadmins having to type the passphrase interactively and they don't have normal user access.
So basically, your workaround is actually the way to install it, just remove the passphrase and make the key and cert readable for the globus user.
The code for myproxy-retrieve is doing something else: if you're root, it assumes you're using host credentials and then it automatically removes the passphrase.

[iWangJiaxiang]

Thank you @msalle .

I managed to launch globus-gatekeeper with non-root user but the gatekeeper service couldn't run jobs.

See log below:

TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 6: Got connection ::ffff:192.168.6.56 at Wed Jun  3 09:54:09 2020

TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 5: Authenticated globus user: /O=Grid/OU=GlobusTest/OU=simpleCA-globusmaster/OU=Globus Simple CA/CN=globususer
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 0: GATEKEEPER_JM_ID 2020-06-03.09:54:09.0000023656.0000000004 for /O=Grid/OU=GlobusTest/OU=simpleCA-globusmaster/OU=Globus Simple CA/CN=globususer on ::ffff:192.168.6.56
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 5: Requested service: jobmanager
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 5: Authorized as local user: gct
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 5: Authorized as local uid: 1002
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 5:           and local gid: 1002
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 0: executing /usr/local/gct6/sbin/globus-job-manager
TIME: Wed Jun  3 09:54:09 2020
 PID: 26934 -- Notice: 0: GRID_SECURITY_HTTP_BODY_FD=11
TIME: Wed Jun  3 09:54:09 2020
 PID: 26935 -- Notice: 0: Set CONTENT_LENGTH=336
TIME: Wed Jun  3 09:54:09 2020
 PID: 26935 -- Notice: 0: Set GATEWAY_INTERFACE to CGI/1.1
TIME: Wed Jun  3 09:54:09 2020
 PID: 26935 -- Notice: 0: Set SERVER_NAME to globusnode
TIME: Wed Jun  3 09:54:09 2020
 PID: 26935 -- Notice: 0: Set SERVER_PORT to 2119
TIME: Wed Jun  3 09:54:10 2020
 PID: 26934 -- Notice: 0: Read 260 bytes from proxy pipe
TIME: Wed Jun  3 09:54:10 2020
 PID: 26934 -- Notice: 0: Child 26935 started
JMA 2020/06/03 09:54:10 GATEKEEPER_JM_ID 2020-06-03.09:54:09.0000023656.0000000004 for /O=Grid/OU=GlobusTest/OU=simpleCA-globusmaster/OU=Globus Simple CA/CN=globususer on ::ffff:192.168.6.56
JMA 2020/06/03 09:54:10 GATEKEEPER_JM_ID 2020-06-03.09:54:09.0000023656.0000000004 mapped to gct (1002, 1002)
JMA 2020/06/03 09:54:10 GATEKEEPER_JM_ID 2020-06-03.09:54:09.0000023656.0000000004 has GRAM_SCRIPT_JOB_ID  manager type fork

The globus-job-manager got stuck at the bottom line.

The output from var/log/globus/gram_gct.log:

ts=2020-06-03T01:59:19.021272Z id=27043 event=gram.job.end level=ERROR gramid=/16866329858697758421/2488345575034030489/ job_status=4 status=-73 reason="the job manager failed to open stdout"

I understand the gatekeeper running as a non-root user could not execute command as another user. However, it seems the gatekeeper even could not execute command as itself.

So the gatekeeper service has to be launched as root user anyway.

[msalle]

Hi,
I was a bit too quick with my previous answer and should have looked a bit further. I now see the confusion. The documentation is actually misleading and could do with some clarification. The text under "Installing from Source Installer" is as far as I can see only relevant for the installation itself. However, the gatekeeper - as you said - will have to run as root, see https://gridcf.org/gct-docs/latest/gram5/admin/index.html#gram5-security-considerations:

The globus-gatekeeper runs as root, and uses its root privilege to access the host’s private key. [...] and then uses the setuid() function to change to that user and execute the globus-job-manager program

The exception is if you run the https://gridcf.org/gct-docs/latest/gram5/pi/index.html#gram5-cmd-globus-personal-gatekeeper which is certainly not what you want to do.

Conclusion: run the gatekeeper as root and make sure the private key for the hostcert has no passphrase.

By the way, I thought you are using the EPEL packages, or do you have problems with the gatekeeper package?

[iWangJiaxiang]

@msalle Thanks for your explain.

The "confusion" showed me the possibility that installing, running GCT as same non-root user works, where all grid users were mapped to that non-root user and all grid jobs were executed as that user.

That's why I tried to compile GCT myself.

If it's true, GCT would be more flexible for less permission need. And all developers need to do is writing the logic that if the gatekeeper is running as non-root user and the grid job wants to be executed as same user, gatekeeper would fork a process and execute job directly instead of using setuid().

---

Since the issue's solved, I just close it. Further communication could go on. :D