Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save images on remote server #3928

Closed
bezkos opened this issue Aug 2, 2019 · 12 comments · Fixed by #3961
Closed

Save images on remote server #3928

bezkos opened this issue Aug 2, 2019 · 12 comments · Fixed by #3961

Comments

@bezkos
Copy link

@bezkos bezkos commented Aug 2, 2019

I want to save images on a remote server via (FTP?). Is this possible with scrapy?
(I know about S3 or Google but i mean in a custom FTP server).

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Aug 5, 2019

@bezkos
Copy link
Author

@bezkos bezkos commented Aug 5, 2019

I dont think scrapy support saving files, images to FTP servers except S3 or Google.
To achieve that, i patched files.py, images.py and pipelines.py and created a new class FTPStore.

@Gallaecio Gallaecio reopened this Aug 5, 2019
@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Aug 5, 2019

Sorry, I missed the ‘images’ bit 🤦‍♂️

@webtekindo
Copy link

@webtekindo webtekindo commented Aug 6, 2019

Hi @bezkos, we are also looking for the same kind of solution, do you mind sharing your class FTPStore ? That will be very helpfull 👍

@bezkos
Copy link
Author

@bezkos bezkos commented Aug 12, 2019

class FTPStore(object):

    FTP_USERNAME = None
    FTP_PASSWORD = None    

    def __init__(self, uri):
        assert uri.startswith('ftp://')
        serverip = uri.split('://', 1)[1]
        self.ftp = FTP()
        # self.ftp.set_debuglevel(2)
        self.ftp.connect(serverip, 21)
        self.ftp.login(self.FTP_USERNAME, self.FTP_PASSWORD)        

    def persist_file(self, path, buf, info, meta=None, headers=None):        
        buf.seek(0)
        self.ftp.storbinary('STOR %s' %path, buf)	
    
    def stat_file(self, path, info):
        Custom code here cause i needed to download images only 1 time without expiration.

        return {'last_modified': last_modified, 'checksum': path}

Hope I helped...

@OmarFarrag
Copy link
Contributor

@OmarFarrag OmarFarrag commented Aug 15, 2019

Working on that

@faizan2700
Copy link
Contributor

@faizan2700 faizan2700 commented Sep 21, 2019

Kindly let me know if someone is still working on this if not please assign this to me. If someone is working on this can you tell me which "good first issues" (or easy ones ) are unclaimed yet I have tried to get one other before and that was already claimed by someone. Thank you in advance.

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Sep 24, 2019

Kindly let me know if someone is still working on this

See #3961

@faizan2700
Copy link
Contributor

@faizan2700 faizan2700 commented Sep 24, 2019

I will study codebase according to that @Gallaecio

@hk1997
Copy link

@hk1997 hk1997 commented Oct 1, 2019

Is someone still working on this?
I would love to contribute

@faizan2700
Copy link
Contributor

@faizan2700 faizan2700 commented Oct 1, 2019

I looked at the data base and tried to understand but for now I am not working on this issue and @Gallaecio will tell you if someone else is workkng on this issue

@Gallaecio
Copy link
Member

@Gallaecio Gallaecio commented Oct 1, 2019

@hk1997 You can see “May be fixed by #3961” at the top of this issue, right under the issue title.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants