New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunked upload in Powershell #14

Closed
christopher-abel opened this Issue Oct 24, 2015 · 8 comments

Comments

Projects
None yet
2 participants
@christopher-abel
Copy link

christopher-abel commented Oct 24, 2015

Hey Hartez,

Great utility, thanks for sharing.

I'm running it inside a Powershell script on Windows Server 2012 R2 to periodically upload backups to dropbox, eg:
C:\ProgramData\chocolatey\bin\pneumatictube.exe -f C:\backups\backup1.zip -p /backups

It works perfectly for small files, but when larger files require 'chunked' upload it always fails with this error msg:

"File is larger than 150MB, using chunked uploading.
An error occurred and your file was not uploaded.
0"

The cmd works as expected in Command Prompt.

Is this a known issue?

Thanks,
Chris

@hartez

This comment has been minimized.

Copy link
Owner

hartez commented Nov 7, 2015

@christopher-abel Is it failing immediately, or does it try to upload the file for a while and then fail?

@christopher-abel

This comment has been minimized.

Copy link

christopher-abel commented Nov 7, 2015

Hi, it starts uploading for a couple of seconds, then fails.

@hartez

This comment has been minimized.

Copy link
Owner

hartez commented Nov 7, 2015

Can you post the PowerShell script you're running it from?

@christopher-abel

This comment has been minimized.

Copy link

christopher-abel commented Nov 8, 2015

Hi Hartez, thanks for getting back to me. This is the basics of the powershell script.. I'm creating a zip file containing a large number of small asset files (mostly jpgs, pngs, pdfs), some of which are nested in sub-dirs. The file name is generated for the current date each time the script runs on a schedule, but I've excluded some of the unrelated details for brevity.

$sourceDir = "C:\PROD_DATA\Assets";
# file name is generated for current date
$filePath = "C:\Backups\Assets\2015_11_01.zip";

# Create Zip Archive
Add-Type -assembly "system.io.compression.filesystem"
[io.compression.zipfile]::CreateFromDirectory($sourceDir, $filePath)

If(Test-Path -Path $filePath)
{
    # Upload current day to Dropbox
    C:\ProgramData\chocolatey\bin\pneumatictube.exe -f $filePath -p "/Backups/Assets"
}

When I run this script in Powershell ISE, it creates the zip and attempts to upload to dropbox (for maybe 2 seconds), then fails with this output:

Uploading 2015_11_01.zip to /Backups/Assets
Ctrl-C to cancel
File is larger than 150MB, using chunked uploading.
An error occurred and your file was not uploaded.
0

If I take just the pneumatictube command, and insert the filepath of the successfully created zip, it runs as expected, uploading in approx 04:41 eg:

C:\ProgramData\chocolatey\bin\pneumatictube.exe -f "C:\Backups\Assets\2015_11_01.zip" -p "/Backups/Assets"
Uploading 2015_11_01.zip to /Backups/Assets
Ctrl-C to cancel
File is larger than 150MB, using chunked uploading.
100% complete.
Whoosh...
Uploaded 2015_11_01.zip to /Backups/Assets; Revision 20835

For now, I've reverted to iterating through my assets dir, and uploading each file individually as they are all well below the 150MB limit. This takes considerably longer.. around 1 hour. It's not a major issue as it runs overnight, but I'd love to be able to use the single zip approach for easier maintenance.

Thanks for your input,
Regards
Chris

@hartez

This comment has been minimized.

Copy link
Owner

hartez commented Nov 9, 2015

@christopher-abel it appears to be an issue with running in ISE. I can create a script similar to yours which runs fine from the console by fails if I'm running under ISE.

Apparently this is because ISE isn't a real Win32 Console. I'll update the code to try and detect whether the console is available and adjust accordingly.

In the meantime, does your script run correctly if you run it directly from the console (instead of in ISE)?

@christopher-abel

This comment has been minimized.

Copy link

christopher-abel commented Nov 11, 2015

@hartez thanks for looking it the issue and getting back to me. Yes my script runs successfully when called from a cmd prompt.

@hartez

This comment has been minimized.

Copy link
Owner

hartez commented Nov 15, 2015

Version 1.2 turns off the progress reporting if it's running under ISE to prevent the crash.

@hartez hartez closed this Nov 15, 2015

@christopher-abel

This comment has been minimized.

Copy link

christopher-abel commented Nov 16, 2015

@hartez Awesome, thanks for update. The new version works great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment