Script to update from release 0.11 or earlier #951

msterin · 2017-02-22T20:59:21Z

Script to update from release 0.11 or earlier (i.e. from the version with Default tenant picking random UUID.). See #942 and #193 for more details.

This script massages DB and moves VMDK to comply with 0.11.1 and later

Steps:
(1) checks that the DB needs upgrade
(2) renames default tenant folder and flips the _DEFAULT symlink
(3) patches DB to use static uuid for default tenant, and patched DB version

Testing: Manual

installed release 0.11
created a few Docker volumes
reinstalled VIB to the latest
checked that docker volume create fails with

docker volume create -d vmdk --name=uv3
Error response from daemon: create uv3: VolumeDriver.Create: DB access error at /etc/vmware/vmdkops/auth-db (
                        Your ESX installation seems to be using configuration DB created by previous
                        version of vSphere Docker Volume Service, and requires upgrade.
                        See https://github.com/vmware/docker-volume-vsphere/blob/master/docs/misc/UpgradeFrom_Pre0.11.1.md for more information.  (_DEFAULT_UUID = d5b5f64a-ff2d-4dac-8327-db1112a367c5, expected = 11111111-1111-1111-1111-111111111111)
                     )

run the script /usr/lib/vmware/vmdkops/bin/vmdkops_post_update.11.1.py
checked the dir structure in dockvols/
created volume again and checked docker runand `docker volume ls.

root@photon-1 [ ~ ]# docker volume create -d vmdk --name=uv3
uv3
root@photon-1 [ ~ ]# docker run -it --rm -v uv3:/uv1 busybox ls /uv1
lost+found

Script output

/usr/lib/vmware/vmdkops/bin/vmdkops_post_update.11.1.py 
Found default tenant: d5b5f64a-ff2d-4dac-8327-db1112a367c5 _DEFAULT This is a default tenant 
Backup file '/etc/vmware/vmdkops/auth-db.bck' already exists - skipping DB backup
Stopping vmdk-opsd service
Stopping vmdkops-opsd with PID=2820968
vmdkops-opsd is not running
Starting conversion of _DEFAULT tenant directory names. old_uid is d5b5f64a-ff2d-4dac-8327-db1112a367c5
Working on Datastore 'datastore1'
('  Info: ', '/vmfs/volumes/datastore1/dockvols/11111111-1111-1111-1111-111111111111', u'/vmfs/volumes/datastore1/dockvols/d5b5f64a-ff2d-4dac-8327-db1112a367c5', '/vmfs/volumes/datastore1/dockvols/_DEFAULT')
  Renaming /vmfs/volumes/datastore1/dockvols/d5b5f64a-ff2d-4dac-8327-db1112a367c5 to /vmfs/volumes/datastore1/dockvols/11111111-1111-1111-1111-111111111111
Working on Datastore 'vsanDatastore'
('  Info: ', '/vmfs/volumes/vsanDatastore/dockvols/11111111-1111-1111-1111-111111111111', u'/vmfs/volumes/vsanDatastore/dockvols/d5b5f64a-ff2d-4dac-8327-db1112a367c5', '/vmfs/volumes/vsanDatastore/dockvols/_DEFAULT')
  Skipping /vmfs/volumes/vsanDatastore/dockvols/d5b5f64a-ff2d-4dac-8327-db1112a367c5 - not found
Working on DB patch...
11111111-1111-1111-1111-111111111111
_DEFAULT
This is a default tenant

Done
Starting vmdk-opsd  service
Starting vmdkops-opsd
vmdkops-opsd is running pid=2821792
watchdog-vsanperfsvc: Terminating watchdog process with PID 2821092
vsanperfsvc started
*** ALL DONE ***

//CC @pdhamdhere @shuklanirdesh82 @lipingxue

lipingxue · 2017-02-22T21:30:42Z

esx_service/cli/vmdkops_post_update.11.1.py

+    """
+
+    tmp_tenant_name = "__tmp_name_upgrade_0_11"
+    sql_query = sql_query_template.format(old_uuid, STATIC_UUID, tmp_tenant_name,


How do we make sure those DB update is in a DB transaction? I think we should try/catch DB errors here.

SQLite3.py automatically wraps any modifying call in a transaction

Those modifying call can throw exception, we should catch and print the exception I think.

How is it different from printing exception and exiting , which is the current behavior in case there is an exception ?

Where do we handle the possible exception by running this "cursor.executescript(sql_query)"?

we don't. My point is that if it happens, python will print the traceback and exit, which is what we want.

ok, got it.

pdhamdhere · 2017-02-22T22:28:41Z

esx_service/cli/vmdkops_post_update.11.1.py

+
+    # STEP : convert dir names to new UUID if needed.
+    print("Starting conversion of _DEFAULT tenant directory names. old_uid is {0}".format(old_uuid))
+    stores = vmdk_utils.get_datastores()


You stopped the service earlier. So, stores is always going to be None right?

No, we do not ask anything from the service since we directly import and use proper modules.

pdhamdhere · 2017-02-22T22:30:22Z

esx_service/cli/vmdkops_post_update.11.1.py

+                print("  Skipping {0} - not found".format(old_dir))
+                continue
+            if os.path.isdir(new_dir):
+                print("  Skipping {0} - {1} already exists".format(old_dir, new_dir))


continue missing?

Yep, good catch !

pdhamdhere

Also include update to https://github.com/vmware/docker-volume-vsphere/blob/master/docs/misc/UpgradeFrom_Pre0.11.1.md

pdhamdhere · 2017-02-22T22:53:59Z

esx_service/cli/vmdkops_post_update.11.1.py

+# Updates ESX with vSphere Docker Volume Service 0.11 (and earlier)
+#  to 0.11.1 and further
+#
+# This code expects to be places in  /usr/lib/vmware/vmdkops/bin/ to pick up correct modules.


Can we check pwd in the begining?

Sure, we can, but I think I just need to drop the comment since the file is now in the bin in the proper place . I'll drop the comment

shuklanirdesh82 · 2017-02-22T23:20:49Z

esx_service/cli/vmdkops_post_update.11.1.py

+
+    if stop_service:
+        print("Stopping vmdk-opsd service")
+        os.system("/etc/init.d/vmdk-opsd stop")


isn't it a good idea to stop service first before initiating backup process?

shuklanirdesh82 · 2017-02-22T23:32:00Z

esx_service/cli/vmdkops_post_update.11.1.py

+    # note that:
+    #       {0} is old_uuid (default tenant uuid pre-upgrade)
+    #       {1} is new_uuid (default tenant uuid post-upgrade)
+    #       {2} is tmp name - we need it to comply with DB constrants


nit: constrants => constraints ?

shuklanirdesh82 · 2017-02-22T23:38:16Z

esx_service/cli/vmdkops_post_update.11.1.py

+        sys.exit(ERROR)
+
+    print("Found default tenant: {0} {1} {2} {3}".format(tenant_id,
+                                                         tenant_name, tenant_desr, tenant_def_db))


nit: can you please rename "tenant_def_db" to "tenant_def_ds" for better reading?

shuklanirdesh82 · 2017-02-22T23:43:16Z

esx_service/cli/vmdkops_post_update.11.1.py

+    cursor.execute("SELECT * from tenants")
+    for r in cursor.fetchone():
+        print(r)
+    print("Done")


curious to know here: don't we need to close() the db connection after finishing migration?

shuklanirdesh82 · 2017-02-23T09:21:57Z

esx_service/cli/vmdkops_post_update.11.1.py

+            if os.path.isdir(new_dir):
+                print("  Skipping {0} - {1} already exists".format(old_dir, new_dir))
+            print("  Renaming {0} to {1}".format(old_dir, new_dir))
+            os.rename(old_dir, new_dir)


while executing this upgrade script on multi-esx testbed, line#131 is failing while invoking on second ESX host.

[root@sc-rdops-vm17-dhcp-15-100:~] python /usr/lib/vmware/vmdkops/bin/script.py Found default tenant: 5be05130-5e18-490b-ba99-42f4b798153d _DEFAULT This is a default tenant on 100 Backing up Config DB to '/etc/vmware/vmdkops/auth-db.bck' Stopping vmdk-opsd service Stopping vmdkops-opsd with PID=1240898 vmdkops-opsd is not running Starting conversion of _DEFAULT tenant directory names. old_uid is 5be05130-5e18-490b-ba99-42f4b798153d Working on Datastore 'sharedVmfs-0' Info: /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111 /vmfs/volumes/sharedVmfs-0/dockvols/5be05130-5e18-490b-ba99-42f4b798153d /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT Skipping /vmfs/volumes/sharedVmfs-0/dockvols/5be05130-5e18-490b-ba99-42f4b798153d - /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111 already exists Renaming /vmfs/volumes/sharedVmfs-0/dockvols/5be05130-5e18-490b-ba99-42f4b798153d to /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111 Traceback (most recent call last): File "/usr/lib/vmware/vmdkops/bin/script.py", line 187, in <module> main() File "/usr/lib/vmware/vmdkops/bin/script.py", line 129, in main os.rename(old_dir, new_dir) OSError: [Errno 39] Directory not empty: '/vmfs/volumes/sharedVmfs-0/dockvols/5be05130-5e18-490b-ba99-42f4b798153d' -> '/vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111' ```

while referring @pdhamdhere's previous comment, I've added missing continue as shown below to get the script working.

With the following code block I got the script working but it tends to data loss i.e. volumes created on shared datastore from docker_host resided on the second ESX (shared vmfs & vsan) are no longer accessible/visible (not sure I have misunderstood the previous comment) .. please have a look at following code block.

IMO, we definitely need to add continue but before releasing the control need to copy existing vmdk data to 111-111...111 folder i.e. _DEFAULT tenant folder.

if not os.path.isdir(old_dir): print(" Skipping {0} - not found".format(old_dir)) continue if os.path.isdir(new_dir): print(" Skipping {0} - {1} already exists".format(old_dir, new_dir)) continue <==== this is modified part as per Prashant's previous comment. print(" Renaming {0} to {1}".format(old_dir, new_dir))

lipingxue

LGTM

msterin · 2017-02-23T23:35:02Z

@shuklanirdesh82 double checked on dual esx/multiple DS data stores and it worked as expected. Merging.

…andom UUID). This script massages DB and moves VMDK to comply with 0.11.1 and later (1) checks that the DB needs upgrade (2) renames default tenant folder and flips the _DEFAULT symlink (or moves files if needed) (3) patches DB to use static uuid for default tenant, and patched DB version

vmwclabot added the cla-not-required label Feb 22, 2017

lipingxue suggested changes Feb 22, 2017

View reviewed changes

pdhamdhere reviewed Feb 22, 2017

View reviewed changes

shuklanirdesh82 reviewed Feb 22, 2017

View reviewed changes

pdhamdhere mentioned this pull request Feb 23, 2017

Does the new "tenant" feature span esxi hosts? #952

Closed

pdhamdhere approved these changes Feb 23, 2017

View reviewed changes

shuklanirdesh82 reviewed Feb 23, 2017

View reviewed changes

msterin force-pushed the script_to_update_default.msterin branch 2 times, most recently from b416028 to fc1b12b Compare February 23, 2017 21:29

lipingxue approved these changes Feb 23, 2017

View reviewed changes

msterin force-pushed the script_to_update_default.msterin branch from fc1b12b to 344c5a7 Compare February 23, 2017 23:37

msterin merged commit 393a96b into master Feb 23, 2017

shuklanirdesh82 deleted the script_to_update_default.msterin branch February 23, 2017 23:50

pdhamdhere added the release-note-needed label Feb 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script to update from release 0.11 or earlier #951

Script to update from release 0.11 or earlier #951

msterin commented Feb 22, 2017 •

edited by shuklanirdesh82

lipingxue Feb 22, 2017

msterin Feb 22, 2017

lipingxue Feb 22, 2017

msterin Feb 22, 2017

lipingxue Feb 22, 2017

msterin Feb 23, 2017

lipingxue Feb 23, 2017

pdhamdhere Feb 22, 2017

msterin Feb 22, 2017

pdhamdhere Feb 22, 2017

msterin Feb 22, 2017

pdhamdhere left a comment

pdhamdhere Feb 22, 2017

msterin Feb 22, 2017

shuklanirdesh82 Feb 22, 2017

shuklanirdesh82 Feb 22, 2017

msterin Feb 23, 2017

shuklanirdesh82 Feb 22, 2017

msterin Feb 23, 2017

shuklanirdesh82 Feb 22, 2017

shuklanirdesh82 Feb 23, 2017

shuklanirdesh82 Feb 23, 2017

lipingxue left a comment

msterin commented Feb 23, 2017

Script to update from release 0.11 or earlier #951

Script to update from release 0.11 or earlier #951

Conversation

msterin commented Feb 22, 2017 • edited by shuklanirdesh82

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdhamdhere left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lipingxue left a comment

Choose a reason for hiding this comment

msterin commented Feb 23, 2017

msterin commented Feb 22, 2017 •

edited by shuklanirdesh82