Permalink
Browse files

Merge remote branch 'upstream/master' into spotinstances

  • Loading branch information...
pauloricardomg committed Aug 2, 2010
2 parents 939faa6 + 0240a90 commit c5cffa36d62e1676c7cc93f1d4e03ede0f51051a
Showing with 560 additions and 363 deletions.
  1. +1 −1 cloud-client/builder/environment.sh
  2. +33 −15 cloud-client/nimbus-cloud-client-src/README.txt
  3. +5 −1 cloud-client/nimbus-cloud-client-src/bin/grid-proxy-init.sh
  4. +2 −2 control/libexec/workspace-control/mount-alter.sh
  5. +36 −23 control/src/python/workspacecontrol/defaults/ImageEditing.py
  6. +28 −8 control/src/python/workspacecontrol/defaults/lvrt/lvrt_common.py
  7. +2 −0 control/src/python/workspacecontrol/mocks/DoNothingPlatform.py
  8. 0 control/var/workspace-control/lock/loopback.lock
  9. +1 −1 cumulus/authz/pynimbusauthz/__init__.py
  10. +2 −2 docs/src/admin/index.html
  11. +1 −1 docs/src/admin/quickstart.html
  12. +2 −2 docs/src/admin/reference.html
  13. +2 −2 docs/src/admin/troubleshooting.html
  14. +2 −2 docs/src/admin/upgrading.html
  15. +111 −5 docs/src/admin/z2c/final-tests.html
  16. +3 −3 docs/src/admin/z2c/index.html
  17. +39 −4 docs/src/admin/z2c/networking-setup.html
  18. +1 −1 docs/src/admin/z2c/service-dependencies.html
  19. +14 −6 docs/src/admin/z2c/service-setup.html
  20. +1 −1 docs/src/admin/z2c/ssh-setup.html
  21. +1 −1 docs/src/admin/z2c/vmm-setup.html
  22. +77 −95 docs/src/changelog.html
  23. +1 −1 docs/src/dev/apis.html
  24. +1 −1 docs/src/dev/docs.html
  25. +15 −2 docs/src/dev/index.html
  26. +9 −2 docs/src/dev/meta.html
  27. +5 −23 docs/src/dev/reference.html
  28. +20 −40 docs/src/dev/releases.html
  29. +1 −1 docs/src/dev/sccs.html
  30. +2 −1 docs/src/faq.html
  31. +105 −88 docs/src/features.html
  32. +2 −2 docs/src/index.html
  33. +5 −5 docs/src/plugins/index.html
  34. +1 −1 docs/src/plugins/service-accounting.html
  35. +1 −1 docs/src/plugins/service-authz.html
  36. +1 −1 docs/src/plugins/service-network.html
  37. +1 −1 docs/src/plugins/service-prop.html
  38. +1 −1 docs/src/plugins/service-reqintake.html
  39. +1 −1 docs/src/plugins/service-sched.html
  40. +1 −1 docs/src/plugins/service-tasks.html
  41. +9 −9 docs/src/summary.html
  42. +4 −0 home/libexec/nimbusconfigure.py
  43. +1 −1 scripts/lib/gt4.0/dist/build.properties
  44. +1 −1 service/service/java/source/share/lib/workspace_service_derby_schema.sql
  45. +1 −1 ...service/java/tests/suites/basic/home/services/share/nimbus/lib/workspace_service_derby_schema.sql
  46. +7 −2 web/bin/run-standalone-ssl.sh
@@ -2,7 +2,7 @@ if [ ! "X$CLCLBUILDER_ENVIRONMENT_DEFINED" = "X" ]; then
return 0
fi
-export CLCLBUILDER_RELEASE_NAME="nimbus-cloud-client-016RC2"
+export CLCLBUILDER_RELEASE_NAME="nimbus-cloud-client-016"
# #########################################################
@@ -40,8 +40,12 @@ The search path the cloud client uses is as follows:
B. proxy
If a normal proxy is present in the /tmp directory and is still valid, that
- is used. This lets the cloud work with all existing certs, tooling, MyProxy,
- etc.
+ is used. This lets the cloud work with all existing certs, tooling,
+ MyProxy, etc.
+
+ Note: if you are using an encrypted key, such as the ones typically
+ provided by grid computing certificate authorities, you will need to
+ generate the proxy mentioned in this step. See the section below.
C. ~/.nimbus/
@@ -51,16 +55,39 @@ The search path the cloud client uses is as follows:
D. ~/.globus/
- Same as #3 but with ~/.globus
+ Same as #3 but with ~/.globus. The key still needs to be unencrypted.
+
+
+2. Test the security setup.
+ $ ./bin/cloud-client.sh --security
-If you want to go the proxy credential route (for example, you have a 'grid'
-certificate), and do not have a proxy credential in place, you can use an embedded
-program to run grid-proxy-init like so:
+3. If you already have a credential and have not given your DN to the cloud
+ administrators, do so sending the distinguished name printed after 'Identity'
+
+
+Encrypted Keys and Proxy Credentials
+------------------------------------
+
+You might need to go the proxy credential route. For example, you were given
+an encrypted certificate, this is typically found in grid computing.
+
+If you do not have a proxy credential in place using some other tool (at for
+example "/tmp/x509up_u1000" where "1000" is your unix account ID number), you
+can use an embedded program to run grid-proxy-init like so:
$ ./bin/grid-proxy-init.sh
+Note that grid-proxy-init does not follow the same search path as the cloud
+client does when the cloud client is looking for unencrypted keys. Instead,
+it only looks for "~/.globus/usercert.pem" and "~/.globus/userkey.pem".
+
+But you can specify the paths exactly if that is not where you keep the cert
+and encrypted key:
+
+ $ ./bin/grid-proxy-init.sh -cert /tmp/usercert.pem -key /tmp/userkey.pem
+
Issues? Try our mailing list and/or run:
$ ./bin/grid-proxy-init.sh -help
@@ -71,15 +98,6 @@ the "lib/certs" directory of the cloud client but you can override like so:
$ export NIMBUS_X509_TRUSTED_CERTS="/path/to/certificates_directory"
-2. Test the security setup.
-
- $ ./bin/cloud-client.sh --security
-
-
-3. If you already have a credential and have not given your DN to the cloud
- administrators, do so sending the distinguished name printed after 'Identity'
-
-
Configuring The Cloud
---------------------
@@ -6,7 +6,11 @@ BASEDIR=`cd $BASEDIR_REL; pwd`
EMBEDDED_GL="$BASEDIR/lib/globus"
EMBEDDED_CADIR="$BASEDIR/lib/certs"
-X509_CERT_DIR="$EMBEDDED_CADIR"
+if [ -n "$NIMBUS_X509_TRUSTED_CERTS" ]; then
+ X509_CERT_DIR="$NIMBUS_X509_TRUSTED_CERTS"
+else
+ X509_CERT_DIR="$EMBEDDED_CADIR"
+fi
export X509_CERT_DIR
OLD_GLOBUS_LOCATION=""
@@ -88,7 +88,7 @@ MOUNT="/bin/mount"
UMOUNT="/bin/umount"
CP="/bin/cp"
-FLOCKFILE=/var/lock/nimbus.mountalter.lock
+FLOCKFILE=/opt/nimbus/var/workspace-control/lock/loopback.lock
FLOCK=/usr/bin/flock
if [ ! -O $FLOCK ]; then
echo "*** can not find flock program, disabling"
@@ -375,4 +375,4 @@ if [ "$DRYRUN" != "true" ]; then
fi
fi
-) 200>>$FLOCKFILE
+) 200<$FLOCKFILE
@@ -412,31 +412,43 @@ def _doMountCopyTasks(self, imagepath, vm_name, mnttask_list, hdimage):
def _doOneMountCopyTask(self, imagepath, src, dst, mntpath, hdimage):
- warning = None
- error = None
if not hdimage:
cmd = "%s %s one %s %s %s %s" % (self.sudo_path, self.mounttool_path, imagepath, mntpath, src, dst)
error = self._doOneMountCopyInnerTask(src, cmd)
+ if error:
+ raise error
+ else:
+ return
+
+ # Some hard disk formats actually mount like partitions, for example
+ # the KVM 'raw' format. We attempt to do partition like mounting
+ # first and then if that fails, try the full blown fdisk + mount
+ # mechanism.
- else:
- # Some hard disk formats actually mount like partitions, for example
- # the KVM 'raw' format. We attempt to do partition like mounting
- # first and then if that fails, try the full blown fdisk + mount
- # mechanism.
- cmd = "%s %s one %s %s %s %s" % (self.sudo_path, self.mounttool_path, imagepath, mntpath, src, dst)
- warning = self._doOneMountCopyInnerTask(src, cmd)
- if warning:
- offsetint = self._guess_offset(imagepath)
- cmd = "%s %s hdone %s %s %s %s %d" % (self.sudo_path, self.mounttool_path, imagepath, mntpath, src, dst, offsetint)
- error = self._doOneMountCopyInnerTask(src, cmd)
+ cmd = "%s %s one %s %s %s %s" % (self.sudo_path, self.mounttool_path, imagepath, mntpath, src, dst)
+ warning = self._doOneMountCopyInnerTask(src, cmd)
+
+ if not warning:
+ # success with partition-style edit
+ return
+ error = None
+ try:
+ offsetint = self._guess_offset(imagepath)
+ cmd = "%s %s hdone %s %s %s %s %d" % (self.sudo_path, self.mounttool_path, imagepath, mntpath, src, dst, offsetint)
+ error = self._doOneMountCopyInnerTask(src, cmd)
+ except Exception,e:
+ error = e
+
+ # warning is always present ('true') at this point
+
if not error:
- return # if there is a warning, it is discarded
- if not warning:
- self.c.log.error(error.msg)
- raise error
- elif warning:
- combined = """
+ # success with HD-image-style edit
+ return
+
+ # error AND warning are present, print both
+
+ combined = """
===========================================================================
Tried multiple methods of mounting the image file.
@@ -454,8 +466,7 @@ def _doOneMountCopyTask(self, imagepath, src, dst, mntpath, hdimage):
%s
===========================================================================
""" % (warning.msg, error.msg)
- self.c.log.error(combined)
- raise IncompatibleEnvironment(combined)
+ raise IncompatibleEnvironment(combined)
def _doOneMountCopyInnerTask(self, src, cmd):
if self.c.dryrun:
@@ -508,7 +519,10 @@ def _guess_offset(self, imagepath):
self.c.log.error(errmsg)
raise IncompatibleEnvironment(errmsg)
- part_pattern = re.compile(r'\n%s.*' % imagepath)
+ # fdisk will truncate the partition name to 79 characters if it's too
+ # long. Match only the first 10 characters of imagepath to detect the
+ # first partition line.
+ part_pattern = re.compile(r'\n%s.*' % imagepath[:10])
lines = []
for m in part_pattern.finditer(output):
lines.append(m.group())
@@ -535,4 +549,3 @@ def _guess_offset(self, imagepath):
self.c.log.debug("offset guess is %d for HD image %s" % (offset, imagepath))
return offset
-
@@ -1,3 +1,5 @@
+import fcntl
+import os
import sys
import zope.interface
import libvirt
@@ -17,6 +19,10 @@ class Platform:
def __init__(self, params, common):
+ self.xen3 = False
+ self.kvm0 = False
+ self.create_flock = False
+
if params == None:
raise ProgrammingError("expecting params")
if common == None:
@@ -34,6 +40,10 @@ def __init__(self, params, common):
self.adapter = lvrt_adapter_xen3.vmmadapter(params, common)
self.intakeadapter = lvrt_adapter_xen3.intakeadapter(params, common)
self.xen3 = True
+ # Because of a race between mount-alter.sh and Xen scripts for
+ # accessing loopback devices, we need to flock the same lock as
+ # mount-alter.sh
+ self.create_flock = True
elif adapter_conf == "kvm0":
self.adapter = lvrt_adapter_kvm0.vmmadapter(params, common)
self.intakeadapter = lvrt_adapter_kvm0.intakeadapter(params, common)
@@ -67,15 +77,26 @@ def create(self, local_file_set, nic_set, kernel):
if self.c.dryrun:
self.c.log.debug("dryrun, not sending")
return
-
+
newvm = None
+ lockfile = None
try:
- newvm = self._vmm().createXML(xml, 0)
- except libvirt.libvirtError,e:
- shorterr = "Problem creating the VM: %s" % str(e)
- self.c.log.error(shorterr)
- self.c.log.exception(e)
- raise UnexpectedError(shorterr)
+ try:
+ if self.create_flock:
+ lockfilepath = self.c.resolve_var_dir("lock/loopback.lock")
+ if not os.path.exists(lockfilepath):
+ raise IncompatibleEnvironment("cannot find lock directory or lock file, make sure lock/loopback.lock exists")
+ lockfile = open(lockfilepath, "r")
+ fcntl.flock(lockfile.fileno(), fcntl.LOCK_EX)
+ newvm = self._vmm().createXML(xml, 0)
+ except libvirt.libvirtError,e:
+ shorterr = "Problem creating the VM: %s" % str(e)
+ self.c.log.error(shorterr)
+ self.c.log.exception(e)
+ raise UnexpectedError(shorterr)
+ finally:
+ if lockfile:
+ lockfile.close()
self.c.log.info("launched '%s'" % newvm.name())
@@ -379,4 +400,3 @@ def _fill_model(self, local_file_set, nic_set, kernel):
self.intakeadapter.fill_model(dom, local_file_set, nic_set, kernel)
return dom
-
@@ -21,6 +21,8 @@ def __init__(self, params, common):
self.p = params
self.c = common
self.persistence = wc_core_persistence.Persistence(self.p, self.c)
+ self.xen3 = False
+ self.kvm0 = False
def _get_mockinfo_expecting(self):
"""Return (name, mockinfo)
@@ -27,7 +27,7 @@
object_types[object_type_gridftp] = 2
object_types[object_type_hdfs] = 3
-Version = "0.1"
+Version = "2.5"
def long_help_callback(option, opt, value, parser, all_opts):
for o in all_opts:
@@ -1,13 +1,13 @@
m4_include(/mcs/m4/worksp.lib.m4)
-_NIMBUS_HEADER(2.5RC2 Administrator Guide)
+_NIMBUS_HEADER(2.5 Administrator Guide)
_NIMBUS_HEADER2(n,n,y,n,n,n,n)
_NIMBUS_LEFT2_COLUMN
_NIMBUS_LEFT2_ADMIN_SIDEBAR(y,n,n,n,n)
_NIMBUS_LEFT2_COLUMN_END
_NIMBUS_CENTER2_COLUMN
_NIMBUS_2_5_DEPRECATED
-<h2>Nimbus 2.5RC2 Admin Guide</h2>
+<h2>Nimbus 2.5 Admin Guide</h2>
<p>
This guide contains configuration information for system administrators
@@ -1,5 +1,5 @@
m4_include(/mcs/m4/worksp.lib.m4)
-_NIMBUS_HEADER(2.5RC2 Admin Installation)
+_NIMBUS_HEADER(2.5 Admin Installation)
_NIMBUS_HEADER2(n,n,y,n,n,n,n)
_NIMBUS_LEFT2_COLUMN
_NIMBUS_LEFT2_ADMIN_SIDEBAR(n,y,n,n,n)
@@ -1,13 +1,13 @@
m4_include(/mcs/m4/worksp.lib.m4)
-_NIMBUS_HEADER(2.5RC2 Admin Reference)
+_NIMBUS_HEADER(2.5 Admin Reference)
_NIMBUS_HEADER2(n,n,y,n,n,n,n)
_NIMBUS_LEFT2_COLUMN
_NIMBUS_LEFT2_ADMIN_SIDEBAR(n,n,y,n,n)
_NIMBUS_LEFT2_COLUMN_END
_NIMBUS_CENTER2_COLUMN
_NIMBUS_2_5_DEPRECATED
-<h2>Nimbus 2.5RC2 Admin Reference</h2>
+<h2>Nimbus 2.5 Admin Reference</h2>
<p>
This section explains some side tasks as well as some
@@ -1,13 +1,13 @@
m4_include(/mcs/m4/worksp.lib.m4)
-_NIMBUS_HEADER(2.5RC2 Troubleshooting)
+_NIMBUS_HEADER(2.5 Troubleshooting)
_NIMBUS_HEADER2(n,n,y,n,n,n,n)
_NIMBUS_LEFT2_COLUMN
_NIMBUS_LEFT2_ADMIN_SIDEBAR(n,n,n,y,n)
_NIMBUS_LEFT2_COLUMN_END
_NIMBUS_CENTER2_COLUMN
_NIMBUS_2_5_DEPRECATED
-<h2>Nimbus 2.5RC2 Troubleshooting</h2>
+<h2>Nimbus 2.5 Troubleshooting</h2>
<p>
Any questions can be posted to the workspace-user mailing list and will
@@ -1,13 +1,13 @@
m4_include(/mcs/m4/worksp.lib.m4)
-_NIMBUS_HEADER(2.5RC2 Upgrade Guide)
+_NIMBUS_HEADER(2.5 Upgrade Guide)
_NIMBUS_HEADER2(n,n,y,n,n,n,n)
_NIMBUS_LEFT2_COLUMN
_NIMBUS_LEFT2_ADMIN_SIDEBAR(n,n,n,n,y)
_NIMBUS_LEFT2_COLUMN_END
_NIMBUS_CENTER2_COLUMN
_NIMBUS_2_5_DEPRECATED
-<h2>Upgrading from Nimbus 2.4 to 2.5RC2</h2>
+<h2>Upgrading from Nimbus 2.4 to 2.5</h2>
<p>
TODO!
Oops, something went wrong.

0 comments on commit c5cffa3

Please sign in to comment.