S3 Feed Export throws boto error: "Connection Reset By Peer"

[samzhang111]

Posted details on SO: http://stackoverflow.com/questions/27131693/scrapyd-s3-feed-export-connection-reset-by-peer

[kmike]

Scrapy uses boto for feed exports, so it is likely a boto issue (the one you've linked at SO, boto/boto#2207). Do you know a workaround?

[jersub]

To send big feed files to S3 and avoid this bug, the fix is to use multipart upload. PR #1559 used to implement this for boto2.

[redapple]

It would be great if someone can resurrect this WIP #1559

[Gallaecio]

Looking at the upstream API, it seems like implementing this change is a matter of:

• Documenting the need to install boto3 for S3 support, rather than botocore.
• Implement code that uploads using boto3’s method instead of botocore to upload files. It seems the interfaces are similar.
• If boto3 is not installed, but botocore is, fall back to the current implementation, and log a deprecation warning.

[Gallaecio]

#4077 is an interesting approach, which starts uploading to S3 right away, rather than storing the whole output on disk first, and then uploading it.

But I would rather have a simpler approach than none. We can always implement the #4077 approach afterwards.

[Gallaecio]

See also the workaround by @ogabrielsantos.

[jazzthief]

I'd like to have a go at the simpler approach, if that's alright.

[jazzthief]

@Gallaecio Hey, I did the "check libraries availability" part, it passes the tests with put_object method. Now I'm working on replacing it with boto3's upload_fileobj - it fails several store tests because it tries to compare a MagicMock attribute to int internally. I've tried to work around it, but looks like the tests need to be altered to support the new method. Am I going in the right direction?

[Gallaecio]

Am I going in the right direction?

It sounds like you are. But feel free to open a draft pull request, and I can have a look.