Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure Blob Storage from SQL Server on Linux #2150

Closed
dahovey opened this issue May 16, 2019 — with docs.microsoft.com · 29 comments
Closed

Azure Blob Storage from SQL Server on Linux #2150

dahovey opened this issue May 16, 2019 — with docs.microsoft.com · 29 comments
Assignees
@VanMSFT
Labels
assigned-to-author doc-bug Feedback about omissions, errors, or confusing or conflicting text in an article. linux/tech Pri2 sql/prod

Comments

@dahovey docs.microsoft.com
Copy link

dahovey commented May 16, 2019

The first paragraph states The backup functionality is the same across platforms. For example, you can backup databases locally, to remote drives, or to Microsoft Azure Blob storage service. Is it really correct that Azure Blob Storage is supported on Linux?

I tried following documentation here https://docs.microsoft.com/en-us/sql/relational-databases/tutorial-sql-server-backup-and-restore-to-azure-blob-storage-service?view=sql-server-2017, but I receive error:

Msg 3292, Level 16, State 6, Line 2
A failure occurred while attempting to execute Backup or Restore with a URL device specified. Consult the operating system error log for details.
Msg 3013, Level 16, State 1, Line 2
BACKUP DATABASE is terminating abnormally.

Using cat /var/opt/mssql/log/errorlog shows (<dbname> replaced):

BACKUP failed to complete the command BACKUP DATABASE &lt;dbname&gt;. Check the backup application log for detailed messages.
2019-05-16 17:44:20.81 spid53      VDI: "C:\binn\BackupToUrl.exe" "b" "p" "680074007400700073003A002F002F0061007A006400610063007500730071006C006200610063006B00750070002E0062006C006F0062002E0063006F00720065002E00770069006E0064006F00770073002E006E00650074002F00700072006F0064002F0045006700670049006E007300700065006300740069006F006E002E00620061006B00" "61007A006400610063007500730071006C006200610063006B0075007000" "01000000D08C9DDF0115D1118C7A00C04FC297EB0100000071DD94E6A43C7E46898CEE110058026F000000001200000061007A007500720065006B00650079000000106600000001000020000000EFE9E128B1ACFB77EBFB5681D9650528430EBF879CE32318425655640A376801000000000E80000000020000200000009DC4790D94DF8E8294B2DE5FDE9BE6246B13CA5F39E57EEB13FF1676E11F804E5000000012C97B1BCEE0912FE948B3B1ADA62324A83B78150A2B9976BC5DE18D52FB4BBEC89B4F46062116E362CFD16286D68682B6018C94C6D06B063157854D5186CE3D6A45EF86386E26D8FAB351EBDA60744240000000ADC39B9047969C4B30BDC84CD2BA56F85AF15CADC3E5CCBF56890FC21D49AAA87FE84131667366A5C0F8BAD85D67418E2FB745B6F63D88D6456F8DE21D338DB4" "NOFORMAT" "4D005300530051004C00530045005200560045005200" "" "DB" "45006700670049006E007300700065006300740069006F006E00" "TRACE"
2019-05-16 17:44:20.82 spid53      BackupToUrl: couldn't load process Error Code:  80070002
2019-05-16 17:44:20.82 Backup      Error: 3041, Severity: 16, State: 1.

Please clarify the statement that Backup to URL (Azure blob storage) is really supported on all platforms.
Running SQL Server 2017 Web on Ubuntu.

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.
@VanMSFT
Copy link
Member

VanMSFT commented May 16, 2019

Hello @dahovey! Thank you for reach out to Microsoft on this Documentation forum. It seems that your question is more of a Support issue, but I'll have my colleague, Mike verify.

@MikeRayMSFT - please help verify that Azure Blob Storage is supported for backup in Linux. Thanks!

@MikeRayMSFT
Copy link
Contributor

Hi @dahovey - thank you for the submission. I think there maybe something we can address with docs. I will get back to you soon.

@visitpay
Copy link

visitpay commented Jun 3, 2019

@MikeRayMSFT Any word on this? Thanks!

@MikeRayMSFT
Copy link
Contributor

I'm putting together an answer for the documentation, but here's what I've got so far:

Backup to page blob isn't supported as it uses older mechanism (using a separate binary, backuptourl.exe). Backup to block blob should work just fine since that method is integrated into engine itself. I'm working on an update for the documentation. We should have an update in ~ 1 week. In the mean time you can try using backup to block blob and it should just work.

Quickstart: SQL Server Backup and Restore to Azure Blob Storage Service

@MikeRayMSFT MikeRayMSFT added the doc-bug Feedback about omissions, errors, or confusing or conflicting text in an article. label Jun 5, 2019
@MikeRayMSFT
Copy link
Contributor

Creating a work item to track 1554601
#please-close

@dahovey
Copy link
Author

dahovey commented Jun 18, 2019

I just updated to CU15 - 14.0.3162.1. Tried again after update, and issue still persists.

@Daemach
Copy link

Daemach commented Jul 17, 2019

I am having this same problem trying to restore from Azure into a linux docker container. Is there a fix coming soon and when?

@jcii jcii mentioned this issue Aug 4, 2019
Closed
@jakedenyer
Copy link

Same issue when trying to restore page blobs to ubuntu linux 16.04 with SQL Server Express 2017.

@MikeRayMSFT mentions

Backup to page blob isn't supported as it uses older mechanism (using a separate binary, backuptourl.exe)

Just curious if this will ever be supported and if there is a timeline for it.

@lukos lukos mentioned this issue Oct 25, 2019
Closed
@rsocol rsocol mentioned this issue Jan 27, 2020
Closed
@nicolaibaralmueller
Copy link

What is the status here. We are testing SQL Server 2017 on Ubuntu in Azure and we are not able to restore from azure storage. As soon as we select the storage container in SSMS an error is returned:
The remote server returned an error: (403) Forbidden.

@MikeRayMSFT
Copy link
Contributor

Reopening this issue because there's still confusion. Will track down straightway.

@MikeRayMSFT MikeRayMSFT reopened this Mar 2, 2020
@MikeRayMSFT
Copy link
Contributor

Thanks all for monitoring this thread - I recognize it's old, but I want to make sure the issues are resolved.

  1. I went through the steps in the tutorial with success on a (Ubuntu 16.04 with SQL Server 2017 from Azure marketplace to a storage account in a different resource group.) My storage account was in the same region with the default configuration. I had the latest (SSMS v 18.4).

  2. Restore was successful. Required the SAS key.

Storage container must be a block blob.

I'm going to keep this issue open for a week or so in case anyone wants to chime in.

We're working on an update to the article to clarify a few things, but at this moment it works as designed.

@Daemach
Copy link

Daemach commented Mar 3, 2020

Are you saying that we can backup to and restore from Azure using SQLS for Linux? I gave up and started using S3 because I could never get it to work. Which tutorial are you referencing?

@MikeRayMSFT
Copy link
Contributor

@Daemach, yes, Backup to Azure using SQL Server for Linux works. The Storage account has to use block blob (not page plobs).

I'm referencing Quickstart: SQL backup and restore to Azure Blob storage service.

I should have called it a Quickstart and not a Tutorial.

@Daemach
Copy link

Daemach commented Mar 4, 2020 via email

@cawrites
Copy link
Contributor

cawrites commented Mar 29, 2020 •
edited

@Daemach @dahovey have you considered https://docs.microsoft.com/en-us/azure/storage/blobs/storage-how-to-mount-container-linux blobfuse

@cawrites
Copy link
Contributor

cawrites commented Apr 3, 2020

@dahovey thank you for contacting Microsoft regarding your question. It has been quite some time since your original request and it appears there are a few suggestions to try if this is still an issue for you. Please don't hesitate to contact us again if you have a question regarding the documentation.

@cawrites
Copy link
Contributor

cawrites commented Apr 3, 2020

#please-close

@HryhoriiDidenko
Copy link

HryhoriiDidenko commented Apr 22, 2020 •
edited

@MikeRayMSFT Hi, I tested this on these versions on docker:
2017-CU20-ubuntu-16.04
2019-CU4-ubuntu-16.04
and have the next errors:

VDI: "C:\binn\BackupToUrl.exe" "b" "p" "680074007400700073003A002F002F006800650074007A006E006500720063006C006F00750064006200690034006D0065002E0062006C006F0062002E0063006F00720065002E00770069006E0064006F00770073002E006E00650074002F00730071006C002D00310063002D00300031002F006500720070002D006300700070002E00620061006B00" "6800650074007A006E006500720063006C006F00750064006200690034006D006500" "0100000000000000000000000000000000000000010000000000000000000000000000000000000023FAC7FAC67455D8FB20FE9CD693074DEB92EC6CC7BF4E2479A083AB1644CB87FF48193D85D17A8079A779651C9E4EBE7EDE59E57EEF22928AC7E29929D684168603C81F1B8228432D615B6FD4218667" "FORMAT" "4D005300530051004C00530045005200560045005200" "" "DB" "7400650073007400" "NOTRACE"
BackupToUrl: couldn't load process Error Code:  80070002
2020-04-22 12:54:07.23 Backup      Error: 3041, Severity: 16, State: 1.
2020-04-22 12:54:07.23 Backup      BACKUP failed to complete the command BACKUP DATABASE test. Check the backup application log for detailed messages.

VDI: "C:\binn\BackupToUrl.exe" "b" "p" "680074007400700073003A002F002F006800650074007A006E006500720063006C006F00750064006200690034006D0065002E0062006C006F0062002E0063006F00720065002E00770069006E0064006F00770073002E006E00650074002F00730071006C002D00310063002D00300031002F006500720070002D006300700070002E00620061006B00" "6800650074007A006E006500720063006C006F00750064006200690034006D006500" "01000000D08C9DDF0115D1118C7A00C04FC297EB01000000EAC005F6570CE84BB0FD58AF68466574000000001200000061007A007500720065006B00650079000000106600000001000020000000030340941878238F79C494C75A98E4C72CE424AC7BD0B5122E277226260D2B77000000000E8000000002000020000000189DFAE10A26935C393ECF9B68168A9CE485937EDF0D1448680062A6BCC6602A500000005095E378BBF31F45B4B4ED9B43C07D4DE13FD0E44D4DF158276B81CCF9C61311BF33FBF9EC8EEAB8B68C4DB72578B36A808CA9939781F6411312C2015ED8112EFBF59CFA360758C697AB23D19835334B4000000001B568690AA99DE0F711A3363E8A2801F29EA6815D8B2CF4148E65557222B677827DF76CBC950E85E7BEA4BA3F0FA7D818DD17270CB4F5ADAEDE8EE10FE464B9" "FORMAT" "4D005300530051004C00530045005200560045005200" "" "DB" "7400650073007400" "NOTRACE"
BackupToUrl: couldn't load process Error Code: 80070002
2020-04-22 13:02:57.21 Backup Error: 3041, Severity: 16, State: 1.
2020-04-22 13:02:57.21 Backup BACKUP failed to complete the command BACKUP DATABASE test. Check the backup application log for detailed messages.

@davidbetz
Copy link

@MikeRayMSFT This needs further clarification. My Linux backup is still looking for "C:\binn\BackupToUrl.exe". That's insane. Worse, your lack of clarity causes readers to try myriad permutations of what you could possible mean. Telling people to use block blobs, not page blobs in incredibly unhelpful since that's not even remotely how that works. At one point you say block blob account, at another you say block blob container (neither is how this works). If you mean ADLS2, say ADLS2, if you mean StorageV2, say StorageV2, if you mean BlobStorage, say BlobStorage. Furthermore, before publishing, verify what does NOT work, then clarify what NOT to do (e.g. do NOT cut the RED wire, but the PURPLE wire). All you're doing is bragging about what works on your machine. Not helpful. I invite you to try again.

@yosbeleg89
Copy link

Any update on this? I'm having the same errors, it seems it's calling the BackupToUrl process in a windows path when using a linux container.

@skrewloose
Copy link

skrewloose commented Oct 3, 2020 •
edited

The lack of clarity is all I am seeing in this thread and the documentation.

Any backup using "WITH CREDENTIAL" uses Page Blob, not supported on Linux. When that is used, it looks for the backuptourl.exe.

Create and use a Shared Access Signature to use block blob.

I was setting up my backups like I had done on Windows and it was not working, looking for the .exe above.

Tested on Ubuntu & SQL 2019 CU6

@yosbeleg89
Copy link

yosbeleg89 commented Oct 5, 2020 •
edited

I tested with SAS and it works (2017-CU21-ubuntu-16.04), it's kind of weird anyway that it is being called C:\binn\BackupToUrl.exe on Linux, instead it should give a more clear error.

@MikeRayMSFT
Copy link
Contributor

@skrewloose - thanks for posting a solution.

@yosbeleg89, @HryhoriiDidenko & @davidbetz, I'm reopning for additional review.

@tejasaks & @VanMSFT - could you please review?

@MikeRayMSFT MikeRayMSFT reopened this Oct 5, 2020
@MikeRayMSFT MikeRayMSFT assigned VanMSFT and unassigned MikeRayMSFT Oct 5, 2020
@VanMSFT
Copy link
Member

VanMSFT commented Oct 5, 2020

SQL Server on Linux only supports backing up to Azure Blob storage using block blobs. Using a storage key for backup and restore will result in a page blog being used, which isn't supported. Use a Shared Access Signature instead. For information on block blogs versus page blogs, see Backup to block blob vs. page blob

@Daemach
Copy link

Daemach commented Oct 5, 2020 via email

@VanMSFT
Copy link
Member

VanMSFT commented Oct 5, 2020

@Daemach - That is already being done. The doc update will publish later today.

@Sossenbinder
Copy link

I just ran into this very problem while attempting to backup a database to url with credential on a linux mssql docker container.

BACKUP DATABASE [TESTDB] TO URL = N'https://<...>.blob.core.windows.net/sqlbackups/TESTDB_202207044.bak' WITH CREDENTIAL = 'DBBackupCred' ,FORMAT ,STATS = 5; GO

and also ran into "A failure occurred while attempting to execute Backup or Restore with a URL device specified. Consult the operating system error log for details.".

Logs of my mssql container point to:

2022-07-04 21:52:18.29 Backup BACKUP failed to complete the command BACKUP DATABASE TESTDB. Check the backup application log for detailed messages. VDI: "C:\binn\BackupToUrl.exe" "b" "p" "680074007400700073003A002F002F006A0075006D0070006500720073006F0063006300750070006100740069006F006E007400720038003500380038002E0062006C006F0062002E0063006F00720065002E00770069006E0064006F00770073002E006E00650074002F00730071006C006200610063006B007500700073002F005400450053005400440042005F003200300032003200300037003000340034002E00620061006B00" "6A0075006D0070006500720073006F0063006300750070006100740069006F006E00740072003800350038003800" "01000000000000000000000000000000000000000100000000000000000000000000000000000000BC41382AAB121E9AF36414AF33F82F70EDCB66675B6E7F115120BD7BCAFCE79F05FAD71F06675497F9782D962F301D97EB74E580BF38B79CD080CCD54F7748D22A0A315B93A7E899BCAF204A1C5EC9DF" "FORMAT" "4D005300530051004C00530045005200560045005200" "" "DB" "540045005300540044004200" "NOTRACE" BackupToUrl: couldn't load process Error Code: 80070002 2022-07-04 21:52:36.91 Backup Error: 3041, Severity: 16, State: 1.

Should this be fixed already?

For verification purposes I installed mssql express on my Windows machine, replicated the SQL database state, and ran the very same query. Works like a charm on windows.

I wonder why the linux container attempts to execute a windows path in the first place?

@tejasaks
Copy link
Contributor

tejasaks commented Jul 5, 2022

@Sossenbinder , Were you using the block blob method as described below (with shared access signature)? If it is trying to use the backuptourl.exe, it means that it was trying to use page blob method for restore which is not supported on Linux. Only backup and restore from block blob are supported.

Backup to block blob vs. page blob

@amvin87 as FYI.

@Sossenbinder
Copy link

Sossenbinder commented Jul 5, 2022 •
edited

@tejasaks

Absolutely, I did accidently use page blob. Thanks for the hint, that helped a lot and works great, I guess I simply missed that point in the docs 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VanMSFT VanMSFT

Labels
assigned-to-author doc-bug Feedback about omissions, errors, or confusing or conflicting text in an article. linux/tech Pri2 sql/prod
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests