Issue 119: download files with different extension than exe #126

Nebelhom · 2013-08-09T13:33:39Z

This pull addresses Issue #119

Not much to it really.

Nebelhom · 2013-08-09T13:34:05Z

Could you review please. Thanks a lot

whimboo · 2013-08-16T08:37:10Z

I tried various combinations with this patch applied and all works great. @Nebelhom please add new tests to the manual list, so we can cover:

zip archive of Firefox on windows
complete.mar for all platforms
tests.zip for all platforms
the status file ´.txt´ for all platforms

One idea I have is that we remove the ´(installer)?´ part from the regex and make it the default value of the extension option. That way it's better understandable. We might want to rename --extension to something better. All that would make it easier for us to get the stub installer part implemented.

For downloading files like ´jsshell-win32.zip´ we should file a separate bug.

Nebelhom · 2013-08-16T11:30:53Z

@whimboo

With regard to the above commit, I have two comments:

I have noticed that quite often the regex_suffix value is the same for all platforms. Is this intended in order to be able to react to potential variations or an artifact of the beginnings of scraper.py? Can I collapse them into one string?

compare:

https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l418 (installer is now gone for win builds)
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l499
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l674

etc.

I am unsure what is a better name for the --extension flag. I am tending towards --suffix but I would be happy for other input, as well.

whimboo · 2013-08-16T12:22:42Z

mozdownload/scraper.py

-                           'win32': 'exe',
-                           'win64': 'exe'}
+                           'win32': r'(\installer.)?exe',
+                           'win64': r'(\installer.)?exe'}


Please do not change those values here! ´installer´ should never be a part of it. This hash only contains real file extensions, which might be overwritten later.

@whimboo

I must have misunderstood you then.

Quote:
"we remove the ´(installer)?´ part from the regex and make it the default value of the extension option."

I assumed you meant the default value for self.extension, which is marked in DEFAULT_FILE_EXTENSIONS

In fact,... where do you want me to move it then?...

Please drop that part given that it doesn't apply anymore to any of your code in this PR. This was related to the other file types which has been spun off as a separate bug. With default I mean that when trying to download a file with a specific extension, the 'installer' string should be a default value for the file type, so that we automatically would grab installers by default. But anyway, it's not related to this pull.

whimboo · 2013-08-16T12:24:42Z

See my other comment regarding the regex_suffix. I don't think that is true.

Regarding the --extension flag we agreed on IRC that we want to move all that to a separate issue. This PR only cares about the real file extension.

Releases don't have zip files, that's correct. So it only applies to daily and tinderbox builds.

davehunt · 2013-08-19T10:16:58Z

tests/test_cases.md

@@ -27,6 +27,7 @@ mozdownload -a firefox -t candidate -p win32 -v 21.0
 mozdownload -a firefox -t candidate -p win32 -v 21.0 -d firefox-candidate-builds
 mozdownload -a firefox -t candidate -p win32 -v 21.0 -l cs
 mozdownload -a firefox -t candidate -p win32 -v 21.0 -l en-GB
+mozdownload -a firefox -t candidate -p win32 -v 21.0 --extension=zip


Nit: Could we move this up a few lines to match the order of the other sections?

Also, please use txt as the extension as it's much quicker for manual testing.

@davehunt

Please check whimboo's comment regarding zip archives of Firefox on Windows ( #126 (comment) )
I assumed, we want to explicitly test for the zip files.

Am I interpreting it correctly?

It depends, really here we're testing that you can specify an alternate extension... we're not really interested in testing that the appropriate files are available for download. If that were to fail it would likely be a release engineering bug and not a mozdownload one.

Nebelhom · 2013-08-21T11:16:13Z

@whimboo @davehunt

I forgot the notification... again :( sorry. Please review :) thx!

whimboo · 2013-08-28T17:04:20Z

This PR has been bitrotted. Can you please update the patch so I can have a quick spot-check for its functionality?

Nebelhom · 2013-08-28T20:45:59Z

@whimboo It should be good to go now...

whimboo · 2013-08-28T21:16:11Z

Sorry, but this does not work:

$ mozdownload -t candidate -v 23.0 -p win32 --extension zip
Retrieving list of candidate builds from https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/23.0-candidates/
Found 1 build: build1
Downloading from: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/23.0-candidates/build1/win32/en-US/firefox-23.0.crashreporter-symbols.zip

It should not download the crash symbols but the build.

Someone similar to daily builds (tinderbox might also be affected)

$ mozdownload -t daily -p win32 --extension zip
mozdownload.scraper.NotFoundError: Folder for builds on 2013-08-28-03-02-02 has not been found: https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2013/08

Why doesn't it find the folder?

I think we have to skip this feature from 1.9 and move it to 1.10.

Nebelhom · 2013-09-24T18:38:04Z

@whimboo

fyi: The reason the crashreporter-symbols.zip is download is, because there is more than one zip file in the folder and crashreporter-symbols happens to be the first in the list.

I'll try to catch you on IRC some time this week and see if we can come up with an elegant solution. Everything I can think of tonight feels hackish.

whimboo · 2013-09-25T10:20:00Z

As of now we should only take zip files which also start with the name of the application specified. So we shouldn't get crashreporter-* files at all.

Nebelhom · 2013-09-25T17:26:56Z

@whimboo:

The binary_regex for ReleaseScraper, ReleaseCandidateScraper and TinderboxScraper ends with .*\.%(EXT)s$.

Note the .* which is the reason for this behaviour. I am sure there is a good reason for this. Do you remember why it was written like this?

The specific lines for binary_regex in the current code are:

https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l442
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l524

whimboo · 2013-10-01T10:10:19Z

So as given on bug 922241 zip archives of Firefox will be removed soon. So would that invalidate this PR?

Nebelhom · 2013-10-01T15:46:05Z

@whimboo

It does not necessarily invalidate the PR. We are allowing for the absence or presence of the 'installer' portion in the filename. Having looked at a tinderbox build we still have the following file formats without the installer part in the name:

firefox-26.0a1.en-US.langpack.xpi
firefox-26.0a1.en-US.win32.checksums
firefox-26.0a1.en-US.win32.checksums.asc
firefox-26.0a1.en-US.win32.json
firefox-26.0a1.en-US.win32.tests.zip
firefox-26.0a1.en-US.win32.txt

These could still be downloaded with mozdownload if that is desireable. If not, we should scrap this PR

whimboo · 2013-10-18T15:51:46Z

Sorry for taking so long with my reply!

It is! At least for the .json and .txt files. Those are not builds but could be of good use for us. So lets the PR updated.

Nebelhom · 2013-10-20T14:47:22Z

@whimboo

I updated it now and changed the manual test cases from zip to json and txt as test.

Apologies for the empty merge commit, but rebasing from the mozilla/mozdownload repository kept on inserting jarekps' last PR of the dailyscraper tests in the middle of my commits. Quite annoying... I hope it is ok like that too

davehunt · 2013-10-21T12:39:30Z

Looks good, and tests well for me. Do you want to have a final look over @whimboo or shall I merge this?

whimboo · 2013-10-22T09:25:29Z

mozdownload/scraper.py

-                        'win32': r'(\.installer)%(STUB)s\.%(EXT)s$',
-                        'win64': r'(\.installer)%(STUB)s\.%(EXT)s$'}
+                        'win32': r'(\.installer)?%(STUB)s\.%(EXT)s$',
+                        'win64': r'(\.installer)?%(STUB)s\.%(EXT)s$'}


So the stub part only exists if we have an installer. So we might want to make the whole part optional?

Any reply here?

oh sorry. I wanted to answer, but must have forgotten. I think it makes sense. Once the tests are in, I will move the stub into the brackets, as well.

Nebelhom · 2013-10-22T14:47:14Z

@whimboo

Your suggestions all make sense. Chronologically, I will push this PR off until #161 is landed, so that I can add the tests above directly to it.

Thanks for the input.

whimboo · 2013-10-24T13:23:11Z

Your suggestions all make sense. Chronologically, I will push this PR off until #161 is landed, so that I can add the tests above directly to it.

Oh, that's most likely better. So disregard my above comment re pull #165. :)

Nebelhom · 2013-11-17T21:51:10Z

@whimboo

I don't know about your current version of mozdownload, but the presence of a stub installer in the win32 version causes all tinderbox_scraper tests with the flag -p win32 to fail.

I need to go and fix Issue #180 first, so that those tests work again, otherwise I can't be certain that my current changes work reliably.

Sorry for the mess...

Nebelhom · 2013-11-18T18:53:42Z

.gitignore

@@ -3,3 +3,29 @@
 build
 dist
 tests/venv
+run_tests.sh~


hmmm... I didn't mean to put this in, but it might as well be. It is a constant source of aggravation for me. If I should remove it let me please know. thanks.

Please remove and adjust your editor to not create those backups for the mozdownload project.

Nebelhom · 2013-11-18T18:59:49Z

Ok, next try. Sorry for the many commits. I forgot to clean up after myself (blush). The tests run through ok and seems ok.

@whimboo: could you please review? Thanks a lot.

whimboo · 2013-11-18T19:58:39Z

...refox/tinderbox-builds/mozilla-central-linux/1374583608/firefox-25.0a1.en-US.linux-i686.json

@@ -0,0 +1,11 @@
+Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nam cursus. Morbi ut mi. Nullam enim leo, egestas id, condimentum at, laoreet mattis, massa. Sed eleifend nonummy diam. Praesent mauris ante, elementum et, bibendum at, posuere sit amet, nibh. Duis tincidunt lectus quis dui viverra vestibulum. Suspendisse vulputate aliquam dui. Nulla elementum dui ut augue. Aliquam vehicula mi at mauris. Maecenas placerat, nisl at consequat rhoncus, sem nunc gravida justo, quis eleifend arcu velit quis lacus. Morbi magna magna, tincidunt a, mattis non, imperdiet vitae, tellus. Sed odio est, auctor ac, sollicitudin in, consequat vitae, orci. Fusce id felis. Vivamus sollicitudin metus eget eros.


I would prefer a .txt file to download. We can make use of the ones in the latest sub folders.

Nebelhom · 2013-11-18T20:50:57Z

@whimboo

next try. thx for your patience

whimboo · 2013-11-18T22:13:52Z

This looks good to me. I will get it merged in a bit.

whimboo · 2013-11-18T22:17:50Z

Landed as 1ca4bbd

This was referenced Aug 16, 2013

Allow the download of any kind of file from the build folder which starts with the application name #128

Open

Allow downloading files like ´jsshell-win32.zip´ #129

Open

whimboo reviewed Aug 16, 2013
View reviewed changes

davehunt reviewed Aug 19, 2013
View reviewed changes

whimboo reviewed Oct 22, 2013
View reviewed changes

whimboo mentioned this pull request Oct 22, 2013

Added unittests for the TinderboxScraper (#161) #170

Closed

Nebelhom mentioned this pull request Nov 17, 2013

Added stub support for TinderboxScraper (#180) #183

Merged

Nebelhom added 2 commits November 18, 2013 19:41

Default win extensions regex changed

3615b54

Revert regex change, add manual test for win32 zip

1484243

Nebelhom added 3 commits November 18, 2013 19:42

Fixed issue in Daily Scraper

189983b

Change manual test for zip extension to txt and json

f08ddc5

Add json test

1d0fd6b

Nebelhom reviewed Nov 18, 2013
View reviewed changes

Add txt test, clean up manifest.ini

8577b17

whimboo reviewed Nov 18, 2013
View reviewed changes

remove json file and test, adjust .gitignore

bb35ee9

whimboo closed this Nov 18, 2013

Nebelhom deleted the issue_119_winextension branch January 12, 2014 16:36

		@@ -0,0 +1,11 @@
		Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nam cursus. Morbi ut mi. Nullam enim leo, egestas id, condimentum at, laoreet mattis, massa. Sed eleifend nonummy diam. Praesent mauris ante, elementum et, bibendum at, posuere sit amet, nibh. Duis tincidunt lectus quis dui viverra vestibulum. Suspendisse vulputate aliquam dui. Nulla elementum dui ut augue. Aliquam vehicula mi at mauris. Maecenas placerat, nisl at consequat rhoncus, sem nunc gravida justo, quis eleifend arcu velit quis lacus. Morbi magna magna, tincidunt a, mattis non, imperdiet vitae, tellus. Sed odio est, auctor ac, sollicitudin in, consequat vitae, orci. Fusce id felis. Vivamus sollicitudin metus eget eros.

Issue 119: download files with different extension than exe #126

Issue 119: download files with different extension than exe #126

Conversation

Nebelhom commented Aug 9, 2013

Nebelhom commented Aug 9, 2013

whimboo commented Aug 16, 2013

Nebelhom commented Aug 16, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whimboo commented Aug 16, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nebelhom commented Aug 21, 2013

whimboo commented Aug 28, 2013

Nebelhom commented Aug 28, 2013

whimboo commented Aug 28, 2013

Nebelhom commented Sep 24, 2013

whimboo commented Sep 25, 2013

Nebelhom commented Sep 25, 2013

whimboo commented Oct 1, 2013

Nebelhom commented Oct 1, 2013

whimboo commented Oct 18, 2013

Nebelhom commented Oct 20, 2013

davehunt commented Oct 21, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nebelhom commented Oct 22, 2013

whimboo commented Oct 24, 2013

Nebelhom commented Nov 17, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nebelhom commented Nov 18, 2013

Choose a reason for hiding this comment

Nebelhom commented Nov 18, 2013

whimboo commented Nov 18, 2013

whimboo commented Nov 18, 2013