What is the proper way to upload a file?

[wongwill86]

Thank you for this library!
What is the proper way to upload a file?

It looks like https://github.com/JuliaParallel/AWS.jl/blob/master/src/S3.jl#L518
should allow specification of data to be of IO.

However it looks like when do_http is called, it passes the IO directly to Requests.put
https://github.com/JuliaParallel/AWS.jl/blob/master/src/S3.jl#L730

The requests library doesn't seem to support uploading files using PUT, only POST with parameter type FileParam https://github.com/JuliaWeb/Requests.jl#file-upload

Is there an easy way to upload files?

[wongwill86]

Hello I have a PR open #71 but It seems that there are a lot of incompatibilities between this library and the required Requests.jl. Is this the correct approach?

[jingpengw]

The upload is not working for me due to this issue.

julia> S3.put_object(env, "znn", "experiments/test.net.h5", f)
ERROR: MethodError: `put` has no method matching put(::URIParser.URI, ::IOStream)
Closest candidates are:
  put(::URIParser.URI)
  put(::URIParser.URI, ::AbstractString)
 in do_http at /usr/people/jingpeng/.julia/v0.4/AWS/src/S3.jl:730
 in do_request at /usr/people/jingpeng/.julia/v0.4/AWS/src/S3.jl:613
 in put_object at /usr/people/jingpeng/.julia/v0.4/AWS/src/S3.jl:511

versioninfo()
Julia Version 0.4.6
Commit 2e358ce (2016-06-19 17:16 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[mdpradeep]

@ jingpengwu, you are passing file stream instead of the contents of the file to this API.

[jingpengw]

thanks for the response. but isn't the put_object function support a IO stream?
https://github.com/JuliaCloud/AWS.jl/blob/master/src/S3.jl#L491

julia>  IOStream <: IO
true

julia> io = IOStream("test", rand(UInt8, 128))
julia> S3.put_object(awsEnv, "znn", "test.dat", io)
ERROR: InexactError()
 in EVP_DigestUpdate at /usr/people/jingpeng/.julia/v0.6/AWS/src/crypto.jl:6 [inlined]
 in md5(::IOStream) at /usr/people/jingpeng/.julia/v0.6/AWS/src/crypto.jl:99
 in do_http(::AWS.AWSEnv, ::AWS.S3.RO) at /usr/people/jingpeng/.julia/v0.6/AWS/src/S3.jl:656
 in #do_request#33(::Bool, ::Function, ::AWS.AWSEnv, ::AWS.S3.RO) at /usr/people/jingpeng/.julia/v0.6/AWS/src/S3.jl:612
 in #put_object#30(::String, ::AWS.S3.PutObjectOptions, ::String, ::Function, ::AWS.AWSEnv, ::String, ::String, ::IOStream) at /usr/people/jingpeng/.julia/v0.6/AWS/src/S3.jl:510
 in put_object(::AWS.AWSEnv, ::String, ::String, ::IOStream) at /usr/people/jingpeng/.julia/v0.6/AWS/src/S3.jl:491

The problem seems to be in md5:
https://github.com/JuliaCloud/AWS.jl/blob/master/src/crypto.jl#L84

For download, I can use file pointer, but not upload!
https://github.com/seung-lab/EMIRT.jl/blob/master/plugins/cloud.jl#L155

could you show what is the proper way to upload a file or an array in memory?

[mattBrzezinski]

Closing this as it is no longer relevant with the newest major release.