Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent behavior with Export-RubrikDatabase #141

Closed
bmcmcm opened this issue Nov 21, 2017 · 8 comments
Closed

Inconsistent behavior with Export-RubrikDatabase #141

bmcmcm opened this issue Nov 21, 2017 · 8 comments
Assignees
@MikeFal
Labels
kind-bug

Comments

@bmcmcm
Copy link

bmcmcm commented Nov 21, 2017 •
edited

Expected Behavior

With the target database dropped from the target SQL server:

 Export-RubrikDatabase -id $db.id -RecoveryDateTime $backupToRecover -FinishRecovery -TargetInstanceId $targetInstanceID -TargetFilePaths $targetfiles -TargetDatabaseName "$($db.name)_EXP" -Confirm:$false -Verbose
                                                                                                                              
VERBOSE: Validate the Rubrik token exists
VERBOSE: Found a Rubrik token for authentication
VERBOSE: Gather API Data for Export-RubrikDatabase
VERBOSE: Load API data for 
VERBOSE: Description: Export MSSQL Database from Rubrik to Destination Instance.
VERBOSE: Build the URI
VERBOSE: URI = https://rubrikprod.somedomain.com/api/v1/mssql/db/MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2/export
VERBOSE: Build the query parameters for 
VERBOSE: Query = Found limit flag
VERBOSE: URI = https://rubrikprod.somedomain.com/api/v1/mssql/db/MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2/export?limit=9999
VERBOSE: Build the body
VERBOSE: Body = {
    "targetFilePaths":  [
                            {
                                "logicalName":  "BIZOPS_DB",
                                "exportPath":  "D:\\Rubrik_Exp\\"
                            },
                            {
                                "logicalName":  "BIZOPS_DB_log",
                                "exportPath":  "D:\\Rubrik_Exp\\"
                            }
                        ],
    "recoveryPoint":  [
                          {
                              "timestampMs":  1511158526000
                          }
                      ],
    "finishRecovery":  true,
    "targetDatabaseName":  "BIZOPS_DB_EXP",
    "targetInstanceId":  "MssqlInstance:::e6ee8ee0-270c-4b52-ad76-aad69b00676f"
}
VERBOSE: Performing the operation "Export MSSQL Database from Rubrik to Destination Instance." on target "MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2".
VERBOSE: Submitting the request
VERBOSE: POST https://rubrikprod.somedomain.com/api/v1/mssql/db/MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2/export?limit=9999 with -1-byte payload
VERBOSE: received 356-byte response of content type application/json; charset=UTF-8
VERBOSE: Formatting return value
VERBOSE: Filter the results

(this is my polling of the async request)
RUNNING 
... 10 second delays
RUNNING 
SUCCEEDED

Current Behavior

With the target database dropped from the target SQL server several minutes later and multiple tries:

 Export-RubrikDatabase -id $db.id -RecoveryDateTime $backupToRecover -FinishRecovery -TargetInstanceId $targetInstanceID -TargetFilePaths $targetfiles -TargetDatabaseName "$($db.name)_EXP" -Confirm:$false -Verbose
Validate the Rubrik token exists
Found a Rubrik token for authentication
Gather API Data for Export-RubrikDatabase
Load API data for 
Description: Export MSSQL Database from Rubrik to Destination Instance.
Build the URI
URI = https://rubrikprod.somedomain.com/api/v1/mssql/db/MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2/export
Build the query parameters for 
Query = Found limit flag
URI = https://rubrikprod.somedomain.com/api/v1/mssql/db/MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2/export?limit=9999
Build the body
Body = {
    "targetFilePaths":  [
                            {
                                "logicalName":  "BIZOPS_DB",
                                "exportPath":  "D:\\Rubrik_Exp\\"
                            },
                            {
                                "logicalName":  "BIZOPS_DB_log",
                                "exportPath":  "D:\\Rubrik_Exp\\"
                            }
                        ],
    "recoveryPoint":  [
                          {
                              "timestampMs":  1511158526000
                          }
                      ],
    "finishRecovery":  true,
    "targetDatabaseName":  "BIZOPS_DB_EXP",
    "targetInstanceId":  "MssqlInstance:::e6ee8ee0-270c-4b52-ad76-aad69b00676f"
}
Performing the operation "Export MSSQL Database from Rubrik to Destination Instance." on target "MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2".
Submitting the request
POST https://rubrikprod.somedomain.com/api/v1/mssql/db/MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2/export?limit=9999 with -1-byte payload
The remote server returned an error: (422) Unprocessable Entity.
FAILED {"message":"Rubrik backup service at 'CSDWQ-BIZSDB1.somedomain.com' returned error: Failed to get disk free space for volume , LastError=The parameter is incorrect.\r\n
(87)","id":"Host.BackupAgentError","params":{"${host}":"CSDWQ-BIZSDB1.somedomain.com","${error}":"Failed to get disk free space for volume , LastError=The parameter is incorrect.\r\n(87)"}}

Failure Information (for bugs)

Verbose examples of success and failure in the examples

Steps to Reproduce

  1. Set the variables required for an export of a database to an alternate SQL server with alternate file paths.
  2. Execute the Export-RubrikDatabase command
  3. Sometimes it works, other times you get: "The remote server returned an error: (422) Unprocessable Entity.", which seems like a rejection of the request by Rubrik, but I don't know what 422 means. I'm not sure what the "Failed to get disk free space for volume" message is about, it seems generic. In all cases the target volume/path has more than 10x the needed space to contain the restored database files

Context

Something I've noticed is that the script might fail repeatedly for several tries, then if I close the powershell session and start a new session, a database that was just failing, suddenly works.

  • Rubrik PowerShell Module Version: 4.0.0.99
  • PowerShell Version: 5.1.14409
  • Operating System: Windows Server 2012R2 STD

Failure Logs

The SQL logs don't show anything when the failure occurs, no request made it that far. If this isn't a powershell module problem, then Rubrik support will have to examine the Rubrik logs to determine what is going wrong.
@MikeFal
Copy link
Contributor

MikeFal commented Nov 21, 2017 via email

@bmcmcm
Copy link
Author

bmcmcm commented Nov 21, 2017

For a single database the GUI or Powershell works fine, except that if I repeat the Powershell script it may fail. I guess I'll open a case with Rubrik, maybe the Rubrik logs will show what is going wrong.

@bmcmcm
Copy link
Author

bmcmcm commented Nov 29, 2017

Some further experimenting has shown that I was able to have a database that was failing to export copied to another source server and the Powershell script successfully exported that backup from the new server to the same target with the same target paths. The source databases were identically configured except for the source path drive letters, and directory names. Rubrik has an open case on this and so far no explanation for the inconsistency.

@bmcmcm
Copy link
Author

bmcmcm commented Dec 7, 2017 •
edited

There now appears to be a missing parameter that is supported by the Rubrik API, but is not included with the Powershell Export-RubrikDatabase command. This seems superfluous, since the export paths are already stated in the targetFilePaths, I assume this is due to some temporary work-around in the API.

How would I best implement "TargetLogFilePath" as seen below in a Powershell command:

id:  MssqlDatabase:::5fedd9a5-2f1a-42b8-9eab-e79151c20f83
 
 
{
     "recoveryPoint":  [
                          {
                              "timestampMs":  1511772414000
                          }
                      ],
  "targetInstanceId": "MssqlInstance:::e6ee8ee0-270c-4b52-ad76-aad69b00676f",
  "targetDatabaseName": "SP_Reporting_EXP",
  "targetLogFilePath": "D:\\Rubrik_Exp\\",
  "targetFilePaths":  [
                            {
                                "logicalName":  "SP_Reporting",
                                "exportPath":  "D:\\Rubrik_Exp\\"
                           },
                            {
                                "logicalName":  "SP_Reporting_1",
                                "exportPath":  "D:\\Rubrik_Exp\\"
                            }
                        ],
  "finishRecovery": true,
  "maxDataStreams": 0
}

@bmcmcm
Copy link
Author

bmcmcm commented Dec 7, 2017 •
edited

I attempted to modify Export-RubrikDatabase by adding:

#Optional Target Log File Path
	[string]$TargetLogFilePath,

And

    Write-Verbose -Message "Build the body"
    $body = @{
      $resources.Body.targetInstanceId = $TargetInstanceId
      $resources.Body.targetDatabaseName = $TargetDatabaseName
      $resources.Body.targetLogFilePath = $TargetLogFilePath
      $resources.Body.finishRecovery = $FinishRecovery.IsPresent

But I get this failure when trying to run it:

DBID: MssqlDatabase:::ad966ae6-2757-46b3-9954-67663bfe91a2
Recovery Date: 2017-12-06T06:21:51Z
TargetID: MssqlInstance:::e6ee8ee0-270c-4b52-ad76-aad69b00676f
System.Collections.DictionaryEntry System.Collections.DictionaryEntry System.Collections.DictionaryEntry System.Collections.DictionaryEntry
ExportName: BIZOPS_DB_EXP
Attempted restoration of BIZOPS_DB_EXP appears to have failed...
A parameter cannot be found that matches parameter name 'TargetLogFilePath'.

@bmcmcm
Copy link
Author

bmcmcm commented Dec 14, 2017

I came up with a hacky work-around using Invoke-RubrikRESTCall instead of trying to modify the module functions with TargetLogFilePath. I would like to understand Get-RubrikAPIData though. It seems like this ought to be dynamically generated to be adaptable to sudden changes in the API as new Rubrik versions are released. Could there be a way to generate $api by querying the Rubrik web instance for the information or is this impractical?

Just trying to modify the static Export-RubrikDatabase section of Get-RubrikAPIData with a TargetLogFilePath parameter seems to fail with an error about TargetLogFilePath being invalid. If I generate my own 'body' with the parameter included, it works OK (but the code looks messy by comparison).

@MikeFal
Copy link
Contributor

MikeFal commented Dec 14, 2017

Trying to make some time to review this. FYI, the targetDataFilePath and targetLogFilePath arguments are to support the Simple mode method for restores. I am looking to implement those arguments.

Something that was discovered recently was a bug with our restore and export process. If an export fails, can you check the sys.master_files entries for the failed restore and see if the logical file names and physical files names match? The way to handle this is to map ALL the files in the database (even ones you might not be moving) and it should work.

In the meantime, I will look at this a little more deeply and look to add the simple method arguments.

@chriswahl chriswahl added this to the v4.0.0 milestone Mar 7, 2018
@MikeFal
Copy link
Contributor

MikeFal commented Apr 9, 2018

Simple method added with #140. File path issue was resolved with Rubrik 4.1.

@MikeFal MikeFal closed this as completed Apr 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MikeFal MikeFal

Labels
kind-bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@chriswahl @MikeFal @bmcmcm