[Veeam Plugin] Can't list Backups

[SadiJr]

ISSUE TYPE

• Enhancement Request

COMPONENT NAME

Veeam Plugin

CLOUDSTACK VERSION

4.15.0.0

OS / ENVIRONMENT

• Windows Server 2019 Datacenter (1809);
• Veeam B&R v11 (build 11.0.0.837), Enterprise license;
• OpenSSH 7.7p1;
• PowerShell 5.1.17763.2090;

SUMMARY

We are validating the use of VMWare hypervisor with Veeam Backup. We followed the step-by-step in the ACS documentation (http://docs.cloudstack.apache.org/en/latest/adminguide/veeam_plugin.html) and backups work fine (take backups, assign or remove VMs to Backup Offerings, import or delete Backup Offerings).

However, listing of backups does not work. I read the code and added some logs to understand if it has any output from PowerShell commands in list backups, and this process is fine:

2021-09-05 21:14:16,875 DEBUG [o.a.c.b.v.VeeamClient] (BackgroundTaskPollManager-6:ctx-82417a48) (logid:069b2990) Veeam response for PowerShell commands [PowerShell Add-PSSnapin VeeamPSSnapin;$backup = Get-VBRBackup -Name "test1-CSBKP-e56b6ef4-c930-4fd0-a952-97f0dde19dbf";if ($backup) { (Get-VBRRestorePoint -Backup:$backup -Name "test1" ^| Where-Object {$_.IsConsistent -eq $true}) }] is: [
VM Name                 Creation Time          Type      
-------                 -------------          ----      
test1                   03/09/2021 23:01:24    Increment 
test1                   04/09/2021 23:01:23    Increment 

Add-PSSnapin : No snap-ins have been registered for Windows PowerShell version 5.
At line:1 char:1
+ Add-PSSnapin VeeamPSSnapin;$backup = Get-VBRBackup -Name test1 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (VeeamPSSnapin:String) [Add-PSSnapin], PSArgumentException
    + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.AddPSSnapinCommand
].

The restore points aren't saved in the table used by ACS (backups). Investigating further, I discovered a insert error in this table:

2021-09-05 21:14:16,880 DEBUG [o.a.c.b.VeeamBackupProvider] (BackgroundTaskPollManager-6:ctx-82417a48) (logid:069b2990) Creating a new entry in backups: [uuid: 4ee1b349-a886-48a0-94db-4822eef4db53, vm_id: 226, external_id: null, type: null, date: null, backup_offering_id: 6, account_id: 9, domain_id: 3, zone_id: 1].
2021-09-05 21:14:16,881 DEBUG [c.c.u.d.T.Transaction] (BackgroundTaskPollManager-6:ctx-82417a48) (logid:069b2990) Rolling back the transaction: Time = 1 Name =  BackgroundTaskPollManager-6; called by -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:95-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:212-$Proxy348.persist:-1-VeeamBackupProvider$1.doInTransactionWithoutResult:299-TransactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:50
2021-09-05 21:14:16,881 ERROR [o.a.c.b.BackupManagerImpl] (BackgroundTaskPollManager-6:ctx-82417a48) (logid:069b2990) Failed to sync backup usage metrics and out-of-band backups due to: [DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: INSERT INTO backups (backups.uuid, backups.vm_id, backups.external_id, backups.type, backups.date, backups.size, backups.protected_size, backups.status, backups.backup_offering_id, backups.account_id, backups.domain_id, backups.zone_id) VALUES (x'34656531623334392D613838362D343861302D393464622D343832326565663464623533', 226, null, null, null, 2546351857664, 4087873952862, 'BackedUp', 6, 9, 3, 1)].
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: INSERT INTO backups (backups.uuid, backups.vm_id, backups.external_id, backups.type, backups.date, backups.size, backups.protected_size, backups.status, backups.backup_offering_id, backups.account_id, backups.domain_id, backups.zone_id) VALUES (x'34656531623334392D613838362D343861302D393464622D343832326565663464623533', 226, null, null, null, 2546351857664, 4087873952862, 'BackedUp', 6, 9, 3, 1)

Apparently this is because ACS expects to receive the output in a specific format, which is different from what we are receiving:

private Backup.RestorePoint getRestorePointFromBlock(String[] parts) {
    String id = null;
    String created = null;
    String type = null;
    for (String part : parts) {
        if (part.matches("Id(\\s)+:(.)*")) {
            String[] split = part.split(":");
            id = split[1].trim();
        } else if (part.matches("CreationTime(\\s)+:(.)*")) {
            String [] split = part.split(":", 2);
            created = split[1].trim();
        } else if (part.matches("Type(\\s)+:(.)*")) {
            String [] split = part.split(":");
            type = split[1].trim();
        }
    }
    return new Backup.RestorePoint(id, created, type);
}

While digging deeper I found out that Veeam, in 11 version, changed from snapin to Module (https://forums.veeam.com/powershell-f26/veeam-11-powershell-snapin-gone-t72598.html, https://forums.veeam.com/powershell-f26/no-snap-ins-have-been-registered-for-windows-powershell-t31603.html, https://www.veeam.com/veeam_backup_11_0_whats_new_wn.pdf). This caused the first line of all SSH commands (PowerShell Add-PSSnapin VeeamPSSnapin) result in error, but this error does not prevent the execution of the command.

Also in 11 Veeam version, it introduced an API to list restore points, so I'll work to fix this behavior in 11 version, keeping old versions working.

[yadvr]

@SadiJr can you test 4.15.1 or 4.15.2RC2?

[yadvr]

@SadiJr on reading the docs your Veeam version isn't supported yet. See:
http://docs.cloudstack.apache.org/en/latest/adminguide/veeam_plugin.html

[SadiJr]

@rhtyd Sorry for the delay in the answer.

I tested both 4.15.1 and 4.15.2RC2. Same error in both. About version 11 not being supported, I had read the documentation previously and noticed that tests were done on version 9.5, but I was hopeful that version would work without problems.

I did some more testing and apparently using Format-List (https://docs.microsoft.com/en-us/powershell/scripting/samples/using-format-commands-to-change-output-view?view=powershell-5.1) solved the problem with minimal code changes. I'll run more tests and, if they pass, open a PR. Also, I opened a PR with my changes of logs improvements, witch help me to identify this problem.

[yadvr]

@SadiJr your other PR to improve veeam logs was merged, does that resolve this issue as well?

[SadiJr]

@rhtyd I tested it at a local lab, and it worked correctly. But I asked other people to test and, as soon as I get positive feedback, I will close this issue.

[SadiJr]

@rhtyd PR #5455 fixed this error, so I am closing this issue.

[nvazquez]

Hi @SadiJr the issue seems to still be present on 4.16.1 (#6465). Do you have a PR for the format-list changes? I could only find the logs improvement PR (#5455)