S3 Multipart Uploads

[lucian1900]

I've started working on this at https://github.com/lucian1900/moto, I figured perhaps it should be tracked here?

[spulec]

Awesome.

[lucian1900]

Ugh, mystery parsing errors. This boto mocking business is hard to debug :)

Traceback (most recent call last):
  File "/Users/Lucian/.virtualenvs/moto/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/Lucian/Prog/moto/moto/core/models.py", line 47, in wrapper
    result = func(*args, **kwargs)
  File "/Users/Lucian/Prog/moto/tests/test_s3/test_s3.py", line 49, in test_multipart_upload
    multipart.complete_upload().should.throw(S3ResponseError)
  File "/Users/Lucian/.virtualenvs/moto/lib/python2.7/site-packages/boto/s3/multipart.py", line 302, in complete_upload
    xml = self.to_xml()
  File "/Users/Lucian/.virtualenvs/moto/lib/python2.7/site-packages/boto/s3/multipart.py", line 156, in to_xml
    for part in self:
  File "/Users/Lucian/.virtualenvs/moto/lib/python2.7/site-packages/boto/s3/multipart.py", line 121, in part_lister
    parts = mpupload.get_all_parts(None, part_number_marker)
  File "/Users/Lucian/.virtualenvs/moto/lib/python2.7/site-packages/boto/s3/multipart.py", line 222, in get_all_parts
    xml.sax.parseString(body, h)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/__init__.py", line 49, in parseString
    parser.parse(inpsrc)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/xmlreader.py", line 125, in parse
    self.close()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/expatreader.py", line 217, in close
    self.feed("", isFinal = 1)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/expatreader.py", line 211, in feed
    self._err_handler.fatalError(exc)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/sax/handler.py", line 38, in fatalError
    raise exception
SAXParseException: <unknown>:1:0: syntax error

[spulec]

Yeah, I'm constantly dropping pdbs inside of boto.

The problem here looks like we need to change the key GET response to return a XML response if there is an 'uploadId' in the querystring. See here. Right now it is returning the value of the key, but boto is trying to parse it as XML.

[lucian1900]

Hmm, I was hoping I'd get away with not implementing part listing.

[kouk]

Hey, just wanted to inform you that I will attempt to continue the work by @lucian1900 here: https://github.com/kouk/moto. So far I just merged all the recent development in @spulec's repo and resolved conflicts so that now we are again right where we were with the previous comments in this issue. Now, to implement part listing... :-)

[lucian1900]

Oh, sorry about that :(

I no longer needed to test multipart uploads, so I gave up on this. I'm so sorry I forgot to say anything about that.

[kouk]

No problem. Thanks for your work!

[kouk]

I have made some progress but I am facing two problems:

1. the test_multipart_upload tries to upload a smaller than allowed file. The API should throw a S3ResponseError. I believe I have done what is necessary for the mock api to do this and indeed I see an S3ResponseError being raised. However the test fails rather than succeeds and I don't know why.
2. with the current version of HTTPPretty the tests fail because of this issue: Python 2.6 POST callbacks get called twice gabrielfalcao/HTTPretty#100. The specific problem is that we cannot calculate the correct ETag for the part because HTTPretty only gives us the last 8192 bytes of data. I have commented in that issue on the workaround that I used. I do not know if it causes any other problems but for me it works and all other tests pass.

If you would like I could send you a pull request, although I would like to find a way to resolve these issues first.

[spulec]

Got it, thanks @kouk. I know @gabrielfalcao is doing some serious work on HTTPretty today and tomorrow and is specifically looking at gabrielfalcao/HTTPretty#100.

[spulec]

Let me know if there is anything I can do to help on this. I know a large refactor of HTTPretty has been completed. May be worth it to open the pull request so others can start reviewing/helping.

[kouk]

with some help from @attili I think we now have completed the work that @lucian1900 started. All tests pass and it's synced with your master branch.