Skip to content

[Veeam] Check for failures in the restore process #7224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 25, 2024
Merged

[Veeam] Check for failures in the restore process #7224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6fe60d6
Validate failure state in Veeam restore process
Feb 14, 2023
993cc04
Address Daan review, and properly call method
JoaoJandre Jun 20, 2024
d432997
Address bryan's reviews
JoaoJandre Jun 20, 2024
d1acd02
remove return
JoaoJandre Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 39 additions & 4 deletions plugins/backup/veeam/src/main/java/org/apache/cloudstack/backup/veeam/VeeamClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ private boolean checkTaskStatus(final HttpResponse response) throws IOException
String type = pair.second();
String path = url.replace(apiURI.toString(), "");
if (type.equals("RestoreSession")) {
return checkIfRestoreSessionFinished(type, path);
checkIfRestoreSessionFinished(type, path);
}
}
return true;
Expand All @@ -361,17 +361,29 @@ private boolean checkTaskStatus(final HttpResponse response) throws IOException
return false;
}

protected boolean checkIfRestoreSessionFinished(String type, String path) throws IOException {
for (int j = 0; j < this.restoreTimeout; j++) {

/**
* Checks the status of the restore session. Checked states are "Success" and "Failure".<br/>
* There is also a timeout defined in the global configuration, backup.plugin.veeam.restore.timeout,<br/>
* that is used to wait for the restore to complete before throwing a {@link CloudRuntimeException}.
*/
protected void checkIfRestoreSessionFinished(String type, String path) throws IOException {
for (int j = 0; j < restoreTimeout; j++) {
HttpResponse relatedResponse = get(path);
RestoreSession session = parseRestoreSessionResponse(relatedResponse);
if (session.getResult().equals("Success")) {
return true;
return;
}

if (session.getResult().equalsIgnoreCase("Failed")) {
String sessionUid = session.getUid();
LOG.error(String.format("Failed to restore backup [%s] of VM [%s] due to [%s].",
sessionUid, session.getVmDisplayName(),
getRestoreVmErrorDescription(StringUtils.substringAfterLast(sessionUid, ":"))));
throw new CloudRuntimeException(String.format("Restore job [%s] failed.", sessionUid));
}
LOG.debug(String.format("Waiting %s seconds, out of a total of %s seconds, for the restore backup process to finish.", j, restoreTimeout));

try {
Thread.sleep(1000);
} catch (InterruptedException ignored) {
Expand Down Expand Up @@ -930,6 +942,29 @@ public Pair<Boolean, String> restoreVMToDifferentLocation(String restorePointId,
return new Pair<>(result.first(), restoreLocation);
}

/**
* Tries to retrieve the error's description of the Veeam restore task that resulted in an error.
* @param uid Session uid in Veeam of the restore process;
* @return the description found in Veeam about the cause of error in the restore process.
*/
protected String getRestoreVmErrorDescription(String uid) {
LOG.debug(String.format("Trying to find the cause of error in the restore process [%s].", uid));
List<String> cmds = Arrays.asList(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this be replaced by REST api ?
https://helpcenter.veeam.com/docs/backup/em_rest/restoresessions_id.html?ver=120
powershell command is slow ..

Copy link
Member

@weizhouapache weizhouapache Dec 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can get the description via powershell command

Restore Type       VM Name              State      Start Time             End Time               Description           
------------       -------              -----      ----------             --------               -----------           
RestoreVm          i-2-226-VM           Stopped    14/12/2023 10:59:56    14/12/2023 11:00:49

the description is empty (probably because the task is done without any issue).

checked rest api reference, there is no description in the respose of restoreSession.
https://helpcenter.veeam.com/docs/backup/em_rest/restoresessions_id.html?ver=120

however, the task has message in its response
https://helpcenter.veeam.com/docs/backup/em_rest/tasks_id.html?ver=120
an example of polledTask when task is done:
{"type":"Task","href":"https://10.0.3.142:9398/api/tasks/task-220","link":[{"href":"https://10.0.3.142:9398/api/tasks/task-220","rel":"Delete"},{"href":"https://10.0.3.142:9398/api/restoreSessions/30e90836-7674-41d7-a16a-a9b5ee0b0a7e?format=Entity","type":"RestoreSession","rel":"Related"}],"task_id":"task-220","state":"Finished","operation":"StartVMRestore","result":{"success":"true","message":"Ok"}}

I wonder if we can use polledTask.getResult().getMessage() instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try to take a look at this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good , thanks @JoaoJandre

String.format("$restoreUid = '%s'", uid),
"$restore = Get-VBRRestoreSession -Id $restoreUid",
"if ($restore) {",
"Write-Output $restore.Description",
"} else {",
"Write-Output 'Cannot find restore session with provided uid $restoreUid'",
"}"
);
Pair<Boolean, String> result = executePowerShellCommands(cmds);
if (result != null && result.first()) {
return result.second();
}
return String.format("Failed to get the description of the failed restore session [%s]. Please contact an administrator.", uid);
}

private boolean isLegacyServer() {
return this.veeamServerVersion != null && (this.veeamServerVersion > 0 && this.veeamServerVersion < 11);
}
Expand Down
40 changes: 39 additions & 1 deletion plugins/backup/veeam/src/test/java/org/apache/cloudstack/backup/veeam/VeeamClientTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ public class VeeamClientTest {
private VeeamClient mockClient;
private static final SimpleDateFormat newDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

private VeeamClient mock = Mockito.mock(VeeamClient.class);

@Rule
public WireMockRule wireMockRule = new WireMockRule(9399);

Expand Down Expand Up @@ -161,7 +163,7 @@ public void checkIfRestoreSessionFinishedTestTimeoutException() throws IOExcepti
Mockito.when(mockClient.get(Mockito.anyString())).thenReturn(httpResponse);
Mockito.when(mockClient.parseRestoreSessionResponse(httpResponse)).thenReturn(restoreSession);
Mockito.when(restoreSession.getResult()).thenReturn("No Success");
Mockito.when(mockClient.checkIfRestoreSessionFinished(Mockito.eq("RestoreTest"), Mockito.eq("any"))).thenCallRealMethod();
Mockito.doCallRealMethod().when(mockClient).checkIfRestoreSessionFinished(Mockito.eq("RestoreTest"), Mockito.eq("any"));
mockClient.checkIfRestoreSessionFinished("RestoreTest", "any");
fail();
} catch (Exception e) {
Expand All @@ -170,6 +172,42 @@ public void checkIfRestoreSessionFinishedTestTimeoutException() throws IOExcepti
Mockito.verify(mockClient, times(10)).get(Mockito.anyString());
}

@Test
public void getRestoreVmErrorDescriptionTestFindErrorDescription() {
Pair<Boolean, String> response = new Pair<>(true, "Example of error description found in Veeam.");
Mockito.when(mock.getRestoreVmErrorDescription("uuid")).thenCallRealMethod();
Mockito.when(mock.executePowerShellCommands(Mockito.any())).thenReturn(response);
String result = mock.getRestoreVmErrorDescription("uuid");
Assert.assertEquals("Example of error description found in Veeam.", result);
}

@Test
public void getRestoreVmErrorDescriptionTestNotFindErrorDescription() {
Pair<Boolean, String> response = new Pair<>(true, "Cannot find restore session with provided uid uuid");
Mockito.when(mock.getRestoreVmErrorDescription("uuid")).thenCallRealMethod();
Mockito.when(mock.executePowerShellCommands(Mockito.any())).thenReturn(response);
String result = mock.getRestoreVmErrorDescription("uuid");
Assert.assertEquals("Cannot find restore session with provided uid uuid", result);
}

@Test
public void getRestoreVmErrorDescriptionTestWhenPowerShellOutputIsNull() {
Mockito.when(mock.getRestoreVmErrorDescription("uuid")).thenCallRealMethod();
Mockito.when(mock.executePowerShellCommands(Mockito.any())).thenReturn(null);
String result = mock.getRestoreVmErrorDescription("uuid");
Assert.assertEquals("Failed to get the description of the failed restore session [uuid]. Please contact an administrator.", result);
}

@Test
public void getRestoreVmErrorDescriptionTestWhenPowerShellOutputIsFalse() {
Pair<Boolean, String> response = new Pair<>(false, null);
Mockito.when(mock.getRestoreVmErrorDescription("uuid")).thenCallRealMethod();
Mockito.when(mock.executePowerShellCommands(Mockito.any())).thenReturn(response);
String result = mock.getRestoreVmErrorDescription("uuid");
Assert.assertEquals("Failed to get the description of the failed restore session [uuid]. Please contact an administrator.", result);
}


private void verifyBackupMetrics(Map<String, Backup.Metric> metrics) {
Assert.assertEquals(2, metrics.size());

Expand Down