CLONE_TARGET not used for non-persistent images in shared and qcow2 TM_MADs

[atodorov-storpool]

Description
Following the oned.conf documentation there are three modes described as follow:

• clone_target : determines how the non persistent images will be cloned when a new VM is instantiated.
  NONE: The image will be linked and no more storage capacity will be used
  SELF: The image will be cloned in the Images datastore
  SYSTEM: The image will be cloned in the System datastore

To Reproduce
There are different and more complex use cases but I think the following is the simplest one

1. Set CLONE_TARGET=SELF in TM_MAD_CONF for "shared"
2. Restart opennebula
3. Register an image in the IMAGE datastore as non-persistent with TM_MAD=shared and attach it to a VM with SYSTEM datastore TM_MAD=shared
4. Check the SYSTEM datastore placeholder for the VM - There is a copy of the file (similar behavior for qcow2 TM_MAD - the qcow snapshot is copied to the disk.X.snap/0 in the SYSTEM datastore)

Expected behavior

A symlink to an image clone for the given datastore created in the IMAGE datastore should be created in the SYSTEM datastore.

Details

• Affected Component: [Storage]
• Hypervisor: [KVM]
• Version: [5.6.x]

Additional context
Looks like that the behavior for the persistent images(LN_TARGET) is hard-coded too always do symlink (and always do a copy if SYSTEM TM_MAD=ssh)

The description of "Shared & Qcow2 Transfer Modes" in Open Cloud Storage Setup is also not clear when a symlink is created and when a file copy for bot persistent and non-persistent images.

I am not sure is a bug or expected behavior that need clarification.

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches

[atodorov-storpool]

I have a feeling that #2246 (comment) is related too...

[atodorov-storpool]

Another affected use-case is the live-migrate when there is Ceph SYSTEM DS and additional shared (NFS,Gluster, etc) IMAGE DS. The live-migrate should work when the shared TM_MAD has CLONE_TARGET=SELF because the Ceph TM_MAD will copy the symlinks created in the VM's SYSTEM DS when doing premigrate. Currently the files of the "shared" images could be corrupted on live-migrate.

[atodorov-storpool]

bump
assuming it is not processed because there are no labels yet

[rsmontero]

Hi Anton

Yes CLONE_TARGET and similar parameters are not configuration parameters. This is, they will not modify the behavior of the driver in anyway. These parameters are used to let know oned how an specific driver perform a given operation.

For example if CLONE_TARGET is self, oned knows that when that driver is used; it needs to take out space from the image datastore and not the system datastore; and so update datastore quotas and usage accordingly.

Now, we have the hybrid modes, in this case we have CLONE_TARGET_SSH for example. This means that when an image datastore is used with a system datastore of type SSH instead of using CLONE_TARGET you need to use the CLONE_TARGET_SSH, because in this case CLONE operations are going to take storage space from the system datastore and not the image datastore.

Hope is not more clear

Cheers!

[rsmontero]

Still not sure if this is a bug or feature or we can close it with the previous notes... I'll leave open (now we have a super-duper stale bot for the catch ;)

[atodorov-storpool]

Hi Ruben,

So if the CLONE_* and LN_* variables are for accounting purposes (and for whitelisting the mixed modes) then there is some sort of inconsistency in the behaviour with the comment in oned.conf:

#   ln_target : determines how the persistent images will be cloned when
#               a new VM is instantiated.
#       NONE: The image will be linked and no more storage capacity will be used
#       SELF: The image will be cloned in the Images datastore
#       SYSTEM: The image will be cloned in the System datastore
#   clone_target : determines how the non persistent images will be
#                  cloned when a new VM is instantiated.
#       NONE: The image will be linked and no more storage capacity will be used
#       SELF: The image will be cloned in the Images datastore
# SYSTEM: The image will be cloned in the System datastore

So the question is how to define the following case?
A VM with SYSTEM datastore TM_MAD=ssh, persistent root disk on IMAGE datastore TM_MAD=shared and volatile disk. The HV has local SSD disk. So when deploying a VM I'd expect to have root disk on NFS and swap on local SSD. So if the host crashes, I'd expect to reschedule the VM on anoteher host and the VM will continue from where the initial host crashed.

I think it is not clear that when the VM is deployed the persistent root disk is not so persistent anymore because it is copied to the host's filesystem.

IMHO clone has simillar flaw - the image is again copied to the SYSTEM DS. so if we have IMAGE and SYSTEM datastores of type shared mounted on NFS it will copy the image from one datastore to the other, taking space in the SYSTEM datastore, despite the CLONE_TARGET definition in the Datastore Template...

In the same time for qcow2 and ceph TM_MADs that are also ssh backed ones(I mean that they are on a local filesystem on the host) the shared TM_MAD behave differentlty and at least the persistent images are with symlinks because of the hard-coded test against TM_MAD=ssh.

[atodorov-storpool]

IMHO the admin should expect the TM_MAD to behave following the description - do nothing, clone/sumlink, copy...

[rsmontero]

I don't see any inconsistency, the parameters determines how the driver perform the operations. These parameters, let say that way are for driver developers, so they can express how the driver works. May be we can change the sentence to:

" ln_target : determines how the driver clones persistent images when a new VM is instantiated."

May it is clearer that way.

A VM with SYSTEM datastore TM_MAD=ssh, persistent root disk on IMAGE datastore TM_MAD=shared and volatile disk.

It would be deployed in the hypervisor local storage (that's the main use case to improve I/O performance). Note that you will have the persistent semantics of SSH. A persistent disk will be copied back from system datastore to image datastore. So no matter the transfer mode, a persistent disk will be always persistent, although the specific semantics may change for different drivers (e.g. when the changes are committed to the original image).

[atodorov-storpool]

A VM with SYSTEM datastore TM_MAD=ssh, persistent root disk on IMAGE datastore TM_MAD=shared and volatile disk.

It would be deployed in the hypervisor local storage (that's the main use case to improve I/O performance). Note that you will have the persistent semantics of SSH. A persistent disk will be copied back from system datastore to image datastore. So no matter the transfer mode, a persistent disk will be always persistent, although the specific semantics may change for different drivers (e.g. when the changes are committed to the original image).

I beg you pardon but there is no "would" in this line :)
My point is that with the current implementation there is no option that alolow the image to stay on shared storage. And it is only with SYSTEM TM_MAD=ssh.

So in the case of a Host failure the VM will lost all data when the VM is recheduled on another host. Because the image will be copied from the "shared" datastore reverting the VM at the state when the image was last saved(last undeploy). IMO the current implementation does not allow the admin to have true persistent image on the shared storage - yes, it is slower than local ssd, but the data is persistent and actual. Even on Host disk failures.

Currently the description of the ln_target in this case is "ln_target: determinaes how the space is accounted by opennebula"

I think that the code should be refactored at least to do symlink by default and do copy on admin/user consent - just like with the ceph tm-mad
does. But the trigger should be other because in both cases the type is file.

Sligthtly different but similar case is with the non-persistent images...

In general, I as an admin would like the driver to follow the CLONE_TARGET as it is described in the oned configuration. When self do a copy on the shared storage and do symlink in SYSTEM ds, when system do copy the image to SYSTEM ds. When none in the case of shared it should rise error "unsupported configuration". Similar with LN_TARGET for the persistent images - when I set NONE - do symlink, when self do rise an error "unsupported by shared tm_mad" and only when system do copy the image to the system datastore.

Hope this clear my concern on the subject :)

Cheers,
Anton

[rsmontero]

I understand, but the parameter is the other way around it does not change the driver behavior but states how the driver works.

When using a shared storage combined with the ssh drivers the ssh drivers will be used. And the ssh driver works by copying the files to the remote storage. The ssh driver doesn't assume a shared storage and that must remain in the same way.

If you need other behavior then we need a different driver set a fs_ssh for example (like fs_lvm that works in a similar way as you described, i.e. assuming a shared storage).

[atodorov-storpool]

So it is not a bug, it's a feature :)

Closing the ticket.

[rsmontero]

:)