New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
get real user name from username and email (for submission as real user with drmaa) #4096
Changes from 3 commits
4f31290
b07ebda
6841faa
d25e552
3a16b36
5ee0d7e
01cffa6
64f5e2e
e197cad
71cd8dd
32d7a30
2ec8054
e81a2ac
c275d25
aa200fe
e8f44de
ae1903d
7fbd3f2
074b90d
f83871d
351adc6
57c1124
698075c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1743,7 +1743,14 @@ def _change_ownership( self, username, gid ): | |
job = self.get_job() | ||
# FIXME: hardcoded path | ||
external_chown_script = self.get_destination_configuration("external_chown_script", None) | ||
cmd = [ '/usr/bin/sudo', '-E', external_chown_script, self.working_directory, username, str( gid ) ] | ||
|
||
cmd = [ '/usr/bin/sudo', '-E', ] | ||
for v in ["PATH", "LD_LIBRARY_PATH", "PKG_CONFIG_PATH"]: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This shouldn't be necessary, as 'sudo -E' does preserve the environment. import os
import subprocess
print('without -E:')
subprocess.call(['sudo', './echo.sh'], shell=False)
print('with -E:')
subprocess.call(['sudo', '-E', './echo.sh'], shell=False) Write a echo.sh script and echo some environment variable only available to the user. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The security policy of our cluster has the env_reset option enabled (see man 5 sudoers) which in particular resets PATH. This seems to be the default on CentOS 6.8. In addition the LD_LIBRARY_PATH and PKG_CONFIG_PATH are reset. I'm not sure if PKG_CONFIG_PATH is actually needed - I just added everything that was removed. I think that the policy of sudo to reset the PATH and LIBRARY_PATH (etc) is actually a good idea. Just imagine a script with sudo rights that calls a binary B oder uses a library L. If you have the possibility to modify the PATH or LIBRARY_PATH before sudoing you have essentially the right to do everything. I think this modification is likely not of interest for many (maybe only me). It could also be that it is completely wrong to solve my problem like this. The question would be how to proceed here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, could you ask your admins to make an exception for the external_chown_script.py ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the PATH we also thought that it is sufficient to hard code the path to the python binary in the script. But then there is still the problem with the python shared object file which is not found. Maybe one of you has an idea? Maybe I could build a special statically linked python binary just for these scripts. Could be a better idea than adding a potential security hole to galaxy. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well, I don't think forcing users to build special version of the python interpreter just for this purpose is great. I kind of like injecting PATH (and LD_LIBRARY_PATH -urgs-, if necessary) instead of propagating all variables with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would it be an option to call
instead of
Then we could just use the python of galaxy's virtual environment if this path is accessible from the code. But this still leaves the problem of the library path (I never understood why the python binary but not the shared object is copied to the venv). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer that this list be configurable, but see also my comment about not defaulting to using sudo at all. |
||
try: | ||
cmd.append('%s=%s'%(v, os.environ[ v ])) | ||
except KeyError: | ||
pass | ||
cmd.extend( [ external_chown_script, self.working_directory, username, str( gid ) ] ) | ||
log.debug( '(%s) Changing ownership of working directory with: %s' % ( job.id, ' '.join( cmd ) ) ) | ||
p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) | ||
# TODO: log stdout/stderr | ||
|
@@ -1770,10 +1777,14 @@ def reclaim_ownership( self ): | |
def user_system_pwent( self ): | ||
if self.__user_system_pwent is None: | ||
job = self.get_job() | ||
try: | ||
self.__user_system_pwent = pwd.getpwnam( job.user.email.split('@')[0] ) | ||
except: | ||
pass | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The extra whitespace here and below isn't needed (sorry for the nitpick). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With the helper/wrapper it would be configurable (at a single place) |
||
# try to get user from email address/username | ||
for cand in [job.user.email.split('@')[0], job.user.username]: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer this to be configurable via a variable in galaxy.ini (sth. like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sounds possible. The question is if this is necessary, since testing both adds only a little computation (I guess its the same as the if-statement that would be necessary when we make this configurable). For me both possibilities would be fine. If we leave it as is I could imagine in case that both ways to determine a user are successful a test checking if they yield the same user would be a good idea. Otherwise: Can somebody point me to examples showing me how to access configuration variables? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm worried about corner cases like users who register an email address whose first part matches another users' username ... who then would get billed for computation time. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For adding options to galaxy.ini you'd write your option in the galaxy.ini.sample (with a bit of documentation) and then you reference it in the config object here. This will then be available as There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK. Will do it this way. |
||
try: | ||
self.__user_system_pwent = pwd.getpwnam( cand ) | ||
except KeyError: | ||
pass | ||
|
||
return self.__user_system_pwent | ||
|
||
@property | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -316,7 +316,15 @@ def stop_job( self, job ): | |
self.ds.kill( ext_id ) | ||
else: | ||
# FIXME: hardcoded path | ||
subprocess.Popen( [ '/usr/bin/sudo', '-E', kill_script, str( ext_id ), str( self.userid ) ], shell=False ) | ||
|
||
command = [ '/usr/bin/sudo', '-E', ] | ||
for v in ["PATH", "LD_LIBRARY_PATH", "PKG_CONFIG_PATH"]: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again, I don't understand why this is necessary, 'sudo -E' should really preserve the environment. |
||
try: | ||
command.append('%s=%s'%(v, os.environ[ v ])) | ||
except KeyError: | ||
pass | ||
command.extend( [ kill_script, str( ext_id ), str( self.userid ) ]) | ||
subprocess.Popen( command, shell=False ) | ||
log.debug( "(%s/%s) Removed from DRM queue at user's request" % ( job.get_id(), ext_id ) ) | ||
except drmaa.InvalidJobException: | ||
log.debug( "(%s/%s) User killed running job, but it was already dead" % ( job.get_id(), ext_id ) ) | ||
|
@@ -362,11 +370,18 @@ def external_runjob(self, external_runjob_script, jobtemplate_filename, username | |
""" | ||
script_parts = external_runjob_script.split() | ||
script = script_parts[0] | ||
command = [ '/usr/bin/sudo', '-E', script] | ||
command = [ '/usr/bin/sudo', '-E', ] | ||
for v in ["PATH", "LD_LIBRARY_PATH", "PKG_CONFIG_PATH"]: | ||
try: | ||
command.append('%s=%s'%(v, os.environ[ v ])) | ||
except KeyError: | ||
pass | ||
command.append( script ) | ||
for script_argument in script_parts[1:]: | ||
command.append(script_argument) | ||
|
||
command.extend( [ str(username), jobtemplate_filename ] ) | ||
command.append( "--assign_all_groups" ) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can add this within the galaxy.ini ( |
||
log.info("Running command %s" % command) | ||
p = subprocess.Popen(command, | ||
shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE) | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -282,8 +282,22 @@ def create_paramfile( trans, uploaded_datasets ): | |
""" | ||
def _chown( path ): | ||
try: | ||
pwent = pwd.getpwnam( trans.user.email.split('@')[0] ) | ||
cmd = [ '/usr/bin/sudo', '-E', trans.app.config.external_chown_script, path, pwent[0], str( pwent[3] ) ] | ||
# get username from email/username | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is a lot of duplication here that should be moved to a single function or method called by all who need it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Indeed. Would the helper script solve this problem already? Otherwise could you be more precise what should be packed in a function (just getting the user or also calling the external scripts). Where would such a function fit? I.e. in which python module? |
||
pwent = None | ||
for cand in [ trans.user.email.split('@')[0], trans.user.username]: | ||
try: | ||
pwent = pwd.getpwnam( cand ) | ||
except KeyError: | ||
pass | ||
assert pwent != None | ||
|
||
cmd = [ '/usr/bin/sudo', '-E', ] | ||
for v in ["PATH", "LD_LIBRARY_PATH", "PKG_CONFIG_PATH"]: | ||
try: | ||
cmd.append('%s=%s'%(v, os.environ[ v ])) | ||
except KeyError: | ||
pass | ||
cmd.extend( [ trans.app.config.external_chown_script, path, pwent[0], str( pwent[3] ) ] ) | ||
log.debug( 'Changing ownership of %s with: %s' % ( path, ' '.join( cmd ) ) ) | ||
p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) | ||
stdout, stderr = p.communicate() | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,14 +1,34 @@ | ||
#!/usr/bin/env python | ||
import os | ||
import os.path | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This import is not required. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK. I removed it. |
||
import sys | ||
|
||
# you may configure the paths below which modifications are allowed. | ||
# should contain the full paths to job_working_directory and new_file_path. | ||
# if set to None every file can be modified by the script. | ||
# this can increase security in particular if write access to this | ||
# script is removed by the admin. | ||
# ALLOWED_PATHS = [ job_working_directory, new_file_path ] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you quote There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
ALLOWED_PATHS = None | ||
|
||
def validate_paramters(): | ||
if len(sys.argv) < 4: | ||
sys.stderr.write("usage: %s path user_name gid\n" % sys.argv[0]) | ||
exit(1) | ||
|
||
path = sys.argv[1] | ||
path = os.path.abspath( sys.argv[1] ) | ||
if ALLOWED_PATHS == None: | ||
allowed = True | ||
else: | ||
allowed = False | ||
for p in ALLOWED_PATHS: | ||
if path.startswith( p ): | ||
allowed = True | ||
break | ||
if not allowed: | ||
sys.stderr.write( "owner and group modifications in %s are not allowed\n" %path ) | ||
sys.exit( 1 ) | ||
|
||
galaxy_user_name = sys.argv[2] | ||
gid = sys.argv[3] | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hardcoded sudo is actually a remnant of the original implementation. Ideally, if one needs sudo it should just be part of the value of the corresponding
*_script
call, at which point you'd have full control over the environment variables passed or not passed.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont get what you are suggesting. We could have a little bash helper/wrapper:
dosudo.sh:
#!/bin/bash
sudo -E AND_WHATEVER_IS_NEEDED "$@"
From python we could then call: dosudo.sh externalhelper.py ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I got it. I moved the sudo calls to galaxy.ini