Globus transfer handling fails to initialize #1

Xarthisius · 2021-05-02T18:57:02Z

Step to reproduce

Register Globus resource
Create a Tale, add globus resources to dataSet
Launch a Tale
cat ../data/<globus_resource>

Expected outcome

See the content of the file

Actual outcome

Despair and tracebacks:

[2021-05-02 18:38:29,560] ERROR: Traceback (most recent call last):
[2021-05-02 18:38:29,561] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/transfer_manager.py", line 24, in run
[2021-05-02 18:38:29,561] ERROR:     self.transferHandler.run()
[2021-05-02 18:38:29,561] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/tm_utils.py", line 34, in run
[2021-05-02 18:38:29,561] ERROR:     self.transfer()
[2021-05-02 18:38:29,562] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 30, in transfer
[2021-05-02 18:38:29,562] ERROR:     self._maybeStartServer()
[2021-05-02 18:38:29,562] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 106, in _maybeStartServer
[2021-05-02 18:38:29,562] ERROR:     self.server.start()
[2021-05-02 18:38:29,563] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 57, in start
[2021-05-02 18:38:29,563] ERROR:     (self.endpointId, self.endpointName) = self._createEndpoint()
[2021-05-02 18:38:29,563] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 223, in _createEndpoint
[2021-05-02 18:38:29,563] ERROR:     _runGCCommand('%s/globusconnectpersonal' % self.gcDir, '-setup', setupKey, '-dir',
[2021-05-02 18:38:29,563] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 19, in _runGCCommand
[2021-05-02 18:38:29,564] ERROR:     raise Exception(msg)
[2021-05-02 18:38:29,564] ERROR: Exception: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-setup', '...', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9') failed with exit code 2:
[2021-05-02 18:38:29,564] ERROR: b'usage: register [-h] [--attributes [file:PATH|JSON_DATA]] [-n NAME]\n                [--description DESCRIPTION] [--owner OWNER] [-z]\n

NOTE

Switching order of args from --setup ... --dir ... to --dir ... --setup ... fixes the issue. Globuspersonalconnect passes everything that comes after --setup to separate routine called register.
Update: then I get a different error:

Exception: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-debug', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9', '-setup', 'e916cf72-97a0-4e49-9f03-3019ae33b3b0') failed with exit code 1: 
b"Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.\nCreating the directory... Done\nrelaytool setup exit with status 1, indicating failure. use debug mode for full output\nrelaytool failed, full error info available in register.log\n\nSetup did not complete successfully.\nYou may want to check /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log for more information\n"

$ cat register.log 
[INFO] cli.py::190 [main] got --setup-key, skip endpoint creation...
[INFO] cli.py::196 [main] starting relaytool setup
[WARNING] relaytool.py::136 [invoke_relaytool] relaytool setup exit with status 1, indicating failure. use debug mode for full output
[INFO] cli.py::200 [main] relaytool stdout:
-----
('',)-----
[INFO] cli.py::201 [main] relaytool stderr:
-----
('',)
-----
[INFO] cli.py::202 [main] full trace of relaytool failure
Traceback (most recent call last):
  File "lib/cli.py", line 198, in main
  File "lib/relaytool.py", line 151, in run_relaytool_setup
  File "lib/relaytool.py", line 146, in invoke_relaytool
lib.relaytool.RelayToolFailureError: ('relaytool setup failed', CompletedProcess(args='/opt/globusconnectpersonal/gt_amd64/bin/relaytool', returncode=1, stdout=b'', stderr=b''))
[ERROR] cli.py::203 [main] relaytool failed, full error info available in register.log

The text was updated successfully, but these errors were encountered:

craig-willis · 2022-02-14T20:04:06Z

I get this error in globusconnectpersonal 3.1.x but not 3.0.4, which we have in our v0.9 image:

docker run -it --user=girder --entrypoint=/bin/bash wholetale/girder:v0.9

/opt/globusconnectpersonal/globusconnectpersonal -debug -setup 29e108db-88ac-48bd-a961-cfca7382ceba -dir /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9
Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.
Creating the directory... Done
Configuration directory: /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta
Contacting relay.globusonline.org:2223
Done!

Xarthisius · 2022-02-14T21:30:00Z

This is really annoying cause when I run this manually:

$ cat ala.py 
import subprocess
import threading

cmd = "/opt/globusconnectpersonal/globusconnectpersonal -dir /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9 -setup 39a06eeb-62e6-41fa-9c6f-3c7ed1514617"

p = subprocess.run(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(p.stdout)

class GCThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self, name="Globus Connect")
        self.daemon = True

    def run(self):
        p = subprocess.run(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        print(p.stdout)

t = GCThread()
t.start()
t.join(timeout=10)

I get

girder@1e556b22a355:~$ python3.9 ala.py 
b'setup completed successfully\n'
b'setup completed successfully\n'

craig-willis · 2022-02-21T15:16:50Z

Re-opening this issue because the problem still exists with GCP 3.1.x and we should try to resolve at some point.

Noting that setup appears to succeed when run directly from the CLI:


$ docker run -it --user girder craigwillis/girder:gcp-3.1.6 bash

girder@81853cdda67c /opt/globusconnectpersonal/globusconnectpersonal -version
Globus Connect Personal 3.1.6

girder@81853cdda67c:/girder$ /opt/globusconnectpersonal/globusconnectpersonal -dir /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9 -setup 52b10576-2690-438f-80c1-f87e74fb42c5
Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.
Creating the directory... Done
setup completed successfully

But fails when run from the globus_handler as above:

[2022-02-21 15:10:56,252] WARNING: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9', '-setup', '52b10576-2690-438f-80c1-f87e74fb42c5') failed with exit code 1:
b"Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.\nCreating the directory... Done\nrelaytool setup exit with status 1, indicating failure. use debug mode for full output\nrelaytool failed, full error info available in register.log\n\nSetup did not complete successfully.\nYou may want to check /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log for more information\n"
b''
[2022-02-21 15:10:56,254] ERROR: Traceback (most recent call last):
[2022-02-21 15:10:56,254] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/transfer_manager.py", line 24, in run
[2022-02-21 15:10:56,254] ERROR:     self.transferHandler.run()
[2022-02-21 15:10:56,255] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/tm_utils.py", line 34, in run
[2022-02-21 15:10:56,255] ERROR:     self.transfer()
[2022-02-21 15:10:56,255] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 30, in transfer
[2022-02-21 15:10:56,256] ERROR:     self._maybeStartServer()
[2022-02-21 15:10:56,256] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 106, in _maybeStartServer
[2022-02-21 15:10:56,256] ERROR:     self.server.start()
[2022-02-21 15:10:56,256] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 57, in start
[2022-02-21 15:10:56,257] ERROR:     (self.endpointId, self.endpointName) = self._createEndpoint()
[2022-02-21 15:10:56,257] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 223, in _createEndpoint
[2022-02-21 15:10:56,257] ERROR:     _runGCCommand('%s/globusconnectpersonal' % self.gcDir, '-dir', self.confDir, '-setup', setupKey)
[2022-02-21 15:10:56,257] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 19, in _runGCCommand
[2022-02-21 15:10:56,258] ERROR:     raise Exception(msg)
[2022-02-21 15:10:56,258] ERROR: Exception: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9', '-setup', '52b10576-2690-438f-80c1-f87e74fb42c5') failed with exit code 1:
[2022-02-21 15:10:56,258] ERROR: b"Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.\nCreating the directory... Done\nrelaytool setup exit with status 1, indicating failure. use debug mode for full output\nrelaytool failed, full error info available in register.log\n\nSetup did not complete successfully.\nYou may want to check /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log for more information\n"
[2022-02-21 15:10:56,258] ERROR: b''

$ cat /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log
[INFO] cli.py::190 [main] got --setup-key, skip endpoint creation...
[INFO] cli.py::196 [main] starting relaytool setup
[WARNING] relaytool.py::136 [invoke_relaytool] relaytool setup exit with status 1, indicating failure. use debug mode for full output
[INFO] cli.py::200 [main] relaytool stdout:
-----
('',)-----
[INFO] cli.py::201 [main] relaytool stderr:
-----
('',)
-----
[INFO] cli.py::202 [main] full trace of relaytool failure
Traceback (most recent call last):
  File "lib/cli.py", line 198, in main
  File "lib/relaytool.py", line 151, in run_relaytool_setup
  File "lib/relaytool.py", line 146, in invoke_relaytool
lib.relaytool.RelayToolFailureError: ('relaytool setup failed', CompletedProcess(args='/opt/globusconnectpersonal/gt_amd64/bin/relaytool', returncode=1, stdout=b'', stderr=b''))
[ERROR] cli.py::203 [main] relaytool failed, full error info available in register.log
[INFO] cli.py::190 [main] got --setup-key, skip endpoint creation...
[INFO] cli.py::196 [main] starting relaytool setup

Xarthisius · 2022-02-21T16:31:43Z

For easy reproducing https://gist.github.com/Xarthisius/79f5fc8b8186d5b9d4340ce5d81c2486

Xarthisius · 2022-02-21T16:40:29Z

Note to self:
This works...

$ docker exec -ti -u girder $(docker ps --filter=name=girder -q) girder-shell
import pathlib
from girder.plugins.globus_handler.clients import Clients
from girder.plugins.globus_handler.server import Server
clients = Clients()
s = Server(clients)
key = s._getEndpointKey()
s.confDir = '%s/.WholeTale/%s' % (pathlib.Path.home(), key)
s._createEndpoint()

Xarthisius transferred this issue from whole-tale/girder_wt_data_manager May 2, 2021

craig-willis mentioned this issue Feb 14, 2022

TransferClient uses kwargs explicitly. Fixes #2 #3

Merged

This was referenced Feb 14, 2022

Pin Globus Connect Personal to v3.0.4 Xarthisius/girder#18

Closed

Pin gcp version to 3.0.4 for now whole-tale/girder#15

Merged

Xarthisius closed this as completed in whole-tale/girder#15 Feb 16, 2022

craig-willis mentioned this issue Feb 18, 2022

Globus transfers fail with error "host endpoint is no longer associated with an active subscription" #4

Closed

craig-willis reopened this Feb 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Globus transfer handling fails to initialize #1

Globus transfer handling fails to initialize #1

Xarthisius commented May 2, 2021 •

edited

Loading

craig-willis commented Feb 14, 2022

Xarthisius commented Feb 14, 2022

craig-willis commented Feb 21, 2022 •

edited

Loading

Xarthisius commented Feb 21, 2022

Xarthisius commented Feb 21, 2022

Globus transfer handling fails to initialize #1

Globus transfer handling fails to initialize #1

Comments

Xarthisius commented May 2, 2021 • edited Loading

Step to reproduce

Expected outcome

Actual outcome

NOTE

craig-willis commented Feb 14, 2022

Xarthisius commented Feb 14, 2022

craig-willis commented Feb 21, 2022 • edited Loading

Xarthisius commented Feb 21, 2022

Xarthisius commented Feb 21, 2022

Xarthisius commented May 2, 2021 •

edited

Loading

craig-willis commented Feb 21, 2022 •

edited

Loading