ftp source support #900

Merged
merged 17 commits into from Dec 1, 2016

Conversation

Projects
None yet
4 participants
Contributor

3v1n0 commented Nov 12, 2016

As requested on bug https://bugs.launchpad.net/snappy/+bug/1602323

This adds support for FTP sources, using urllib was the quickest choice, and with some refactor of the indicator module, adding a progressbar was possible too. If using ftplib or others is preferred let me know, but I went for the fastest and safer way.

In theory this path could be used also for http(s) downloads, but not to change what we've done so far, I've left that job to the nicer requests library. requests being an HTTP only library has no native suppot for this, while trying to use something like requests-ftp I don't think is a choice (looking quite immature).

Contributor

3v1n0 commented Nov 16, 2016

This will probably need to be fixed once #906 gets merged...

Contributor

3v1n0 commented Nov 16, 2016

Wondering why the integration test fails in autopkgtest with the urllib.error.URLError: <urlopen error ftp error: TimeoutError(110, 'Connection timed out')> error, while it works fine here and in travis...

@3v1n0 3v1n0 referenced this pull request in ubuntu-core/snappy-jenkins Nov 18, 2016

Open

Export ftp_proxy on autopkgtests #210

Contributor

3v1n0 commented Nov 18, 2016

Ok the connection timeout issue was due to missing ftp_proxy in autopkgtest...

For it to pass, right now it was enough to reuse the http_proxy, unfortunately changing it in debian/tests wasn't enough, so I went for this workaround: eb078e8.

That can be removed once ubuntu-core/snappy-jenkins#210 will be applied, I guess.

@elopio if you can give a look to both, it would be appreciated ;-)

elopio approved these changes Nov 18, 2016

I left some minor naming comments, but I'll leave my +1 here anyway.

snapcraft/internal/indicators.py
@@ -64,6 +71,36 @@ def download_requests_stream(request_stream, destination, message=None):
progress_bar.finish()
+class UrllibDownlaoder(object):
@elopio

elopio Nov 18, 2016

Member

typo: Downlaoder/Downloader

snapcraft/internal/indicators.py
+ if self.progress_bar:
+ self.progress_bar.finish()
+
+ def _progress_cb(self, block_num, block_size, total_length):
@elopio

elopio Nov 18, 2016

Member

I know I'm being a pita, but can you rename this to _progress_callback?
I have a hardtime with acronyms because my brain is not yet fast enough in english to understand them. I always have to spent a few minutes parsing it. That's the only thing I hate about go, and that's why I love that python code tends to be verbose :)

snapcraft/tests/__init__.py
@@ -109,6 +111,22 @@ def setUp(self):
self.useFixture(fixture_setup.FakeParts())
+class FakeFileHttpServerBasedTestCase(TestCase):
@elopio

elopio Nov 18, 2016

Member

The python style is to name it FakeFileHTTPServerBasedTestCase

Member

elopio commented Nov 18, 2016

@3v1n0 that sounds like a thing to add to our autopkgtest setup. Can you talk to pitti about it?
I'm ok with the workaround, and thanks for adding this feature.

Contributor

3v1n0 commented Nov 18, 2016

Ok @elopio, changed the naming as suggested.

As for the ftp_proxy thing, isn't ubuntu-core/snappy-jenkins#210 enough?!?

Canonical datacenter's proxy does not allow ftp, sorry. You have to use http, or skip the test.

Member

elopio commented Nov 18, 2016

@3v1n0 nop, sorry, I forgot to reply to that. Last week we stopped using that jenkins server, and instead we are now sending the PRs directly to the autopkgtest infrastructure: https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure#Integration_with_GitHub_and_GitLab_pull.2Fmerge_requests

Contributor

3v1n0 commented Nov 18, 2016

@martinpitt but... If I set the ftp_proxy=$http_proxy the test does pass... So why not just doing that by default?

Contributor

3v1n0 commented Nov 21, 2016

@sergiusens any moment for looking at this?

3v1n0 added some commits Nov 12, 2016

sources: add support for ftp sources using urllib
if a ftp source is required we fallback to urllib,
still using the fancy progress bar. (LP: #1602323)
test_sources: add more tests for url-opener
Be more conservative on progress-bar in indicator
indicators: bind progress_bar to callback using a lambda
The static attribute was nice, but too hacky too :-P
test_sources: create base test case for HTTP tests
And move FakeFileHTTPRequestHandler server to fake_servers
Contributor

3v1n0 commented Nov 30, 2016

Rebased on latest master

If requests-ftp where available on the distro I would of told you to use that. Nice work.

@sergiusens sergiusens merged commit eba7dd2 into snapcore:master Dec 1, 2016

5 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.06%) to 98.229%
Details
xenial-amd64 autopkgtest finished (success)
Details
yakkety-amd64 autopkgtest finished (success)
Details
zesty-amd64 autopkgtest finished (success)
Details

kalikiana pushed a commit to kalikiana/snapcraft that referenced this pull request Apr 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment