B2G nightly dirs started proliferating recently, with many such dirs for a given date, most of which don't contain a build. See for example:
Which has this bevy of dirs for 2012-12-10:
2012-12-10-12-20-53-mozilla-beta/ 10-Dec-2012 12:27
2012-12-10-12-23-21-mozilla-beta/ 10-Dec-2012 12:26
2012-12-10-12-23-28-mozilla-beta/ 10-Dec-2012 12:37
2012-12-10-12-23-36-mozilla-beta/ 10-Dec-2012 12:26
2012-12-10-12-23-47-mozilla-beta/ 10-Dec-2012 12:27
2012-12-10-12-24-20-mozilla-beta/ 10-Dec-2012 12:34
2012-12-10-12-38-57-mozilla-beta/ 10-Dec-2012 14:42
2012-12-10-12-42-07-mozilla-beta/ 10-Dec-2012 14:52
2012-12-10-12-43-24-mozilla-beta/ 10-Dec-2012 13:25
2012-12-10-12-43-46-mozilla-beta/ 10-Dec-2012 14:49
2012-12-10-12-43-59-mozilla-beta/ 10-Dec-2012 15:03
2012-12-10-12-46-54-mozilla-beta/ 10-Dec-2012 13:52
2012-12-10-12-47-10-mozilla-beta/ 10-Dec-2012 13:54
2012-12-10-12-47-18-mozilla-beta/ 10-Dec-2012 14:02
2012-12-10-12-47-19-mozilla-beta/ 10-Dec-2012 14:02
2012-12-10-13-37-54-mozilla-central/ 10-Dec-2012 14:17
2012-12-10-13-37-55-mozilla-central/ 10-Dec-2012 14:18
2012-12-10-14-57-04-mozilla-beta/ 10-Dec-2012 15:36
2012-12-10-14-58-42-mozilla-beta/ 10-Dec-2012 15:53
2012-12-10-14-58-58-mozilla-beta/ 10-Dec-2012 15:49
2012-12-10-14-59-43-mozilla-beta/ 10-Dec-2012 15:51
2012-12-10-15-03-46-mozilla-beta/ 10-Dec-2012 16:07
2012-12-10-15-04-04-mozilla-beta/ 10-Dec-2012 17:06
2012-12-10-15-04-06-mozilla-beta/ 10-Dec-2012 17:05
2012-12-10-20-12-30-mozilla-beta/ 10-Dec-2012 20:28
Finding a build thus requires a more sophisticated algorithm than the one that mozdownload currently uses. It isn't enough to find all dirs for the given day/channel and then pick the last one. mozdownload must also filter the dirs by whether or not they contain an actual build.
Here's a branch that does that. Along the way, it fixes a recent regression that triggers an exception when checking if tmp_file is a file under certain circumstances in which tmp_file is undefined. And it enhances the directory parser to allow filtering by both regex and function.
don't assume tmp_file is defined
add filtration by function in addition to regex
filter potential build dirs by whether or not they contain a build
even if a time has been included in a date for which to get build inf…
…o, don't assume a build necessarily exists at that time
don't assume non-en-US build dirs have -l10n appended to their names …
…(as B2G's 'multi' localization build dirs do not)
Revert "even if a time has been included in a date for which to get b…
…uild info, don't assume a build necessarily exists at that time"
This reverts commit 9dd643d.
when retrieving the build status file for the latest build, make sure…
… we get the correct file for the locale
I found two more problems with the scraper code while trying to get the latest B2G build from the mozilla-b2g18 branch today.
The latest-mozilla-b2g18/ dir has two text files with info on Mac builds, b2g-18.0.en-US.mac64.txt, which points to the 2012-12-13-03-03-51-mozilla-b2g18/ directory; and b2g-18.0.multi.mac64.txt, which points to the 2012-12-13-09-53-15-mozilla-b2g18/ directory.
The first problem was that DailyScraper wouldn't necessarily get the right build status file when retrieving the latest build, since it didn't take the locale into account when retrieving the file; and later that would cause problems when it tried to download the build, since then it does take the locale into account.
I first fixed this by ignoring a mismatch between the date/time in the status file and the date/time of the build dir. But then I realized that the mismatch masked the real problem and reverted that change in favor of making sure DailyScraper retrieves the right build status file for the given locale.
The second problem was that get_build_info_for_date() assumes that build dirs for non-en-US builds have -l10n appended to their names, but the ones for B2G don't have that suffix. So I fixed this by making the suffix optional in the regex that identifies build dirs.
don't require Windows builds to have '.installer' in their names
@whimboo: The last commit reverts a change you made in pull 31 that broke downloading of Windows B2G builds, which don't have an installer variant. See for example this directory containing the most recent Windows build:
Presumably you made the change for a reason, though, and I don't want to break that other use case! Perhaps you can elaborate, and then we can figure out how to keep both working?
@mykmelez, can you please put the last commit into a separate pull? It's not really related to that one and also a regression. Sorry for that, we need tests FWIW.
Not sure when I will find the time to check this pull request. I'm on PTO starting tomorrow. How critical is that fix for you?
@whimboo, no problem, I'll put the last commit into a separate pull.
And this fix isn't critical, as we can use my fork of mozdownload in the meantime. So don't worry about checking it today; let's pick it back up when you return from PTO!
Revert "don't require Windows builds to have '.installer' in their na…
This reverts commit 5b3267e.
I've rebased and updated the changes from this pull and opened a new pull request as #99