filter potential build dirs by whether or not they contain a build #47

Closed
wants to merge 9 commits into from

3 participants

@mykmelez
Mozilla member

B2G nightly dirs started proliferating recently, with many such dirs for a given date, most of which don't contain a build. See for example:

https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2012/12/

Which has this bevy of dirs for 2012-12-10:

  2012-12-10-12-20-53-mozilla-beta/ 10-Dec-2012 12:27
  2012-12-10-12-23-21-mozilla-beta/ 10-Dec-2012 12:26
  2012-12-10-12-23-28-mozilla-beta/ 10-Dec-2012 12:37
  2012-12-10-12-23-36-mozilla-beta/ 10-Dec-2012 12:26
  2012-12-10-12-23-47-mozilla-beta/ 10-Dec-2012 12:27
  2012-12-10-12-24-20-mozilla-beta/ 10-Dec-2012 12:34
  2012-12-10-12-38-57-mozilla-beta/ 10-Dec-2012 14:42
  2012-12-10-12-42-07-mozilla-beta/ 10-Dec-2012 14:52
  2012-12-10-12-43-24-mozilla-beta/ 10-Dec-2012 13:25
  2012-12-10-12-43-46-mozilla-beta/ 10-Dec-2012 14:49
  2012-12-10-12-43-59-mozilla-beta/ 10-Dec-2012 15:03
  2012-12-10-12-46-54-mozilla-beta/ 10-Dec-2012 13:52
  2012-12-10-12-47-10-mozilla-beta/ 10-Dec-2012 13:54
  2012-12-10-12-47-18-mozilla-beta/ 10-Dec-2012 14:02
  2012-12-10-12-47-19-mozilla-beta/ 10-Dec-2012 14:02
  2012-12-10-13-37-54-mozilla-central/  10-Dec-2012 14:17
  2012-12-10-13-37-55-mozilla-central/  10-Dec-2012 14:18
  2012-12-10-14-57-04-mozilla-beta/ 10-Dec-2012 15:36
  2012-12-10-14-58-42-mozilla-beta/ 10-Dec-2012 15:53
  2012-12-10-14-58-58-mozilla-beta/ 10-Dec-2012 15:49
  2012-12-10-14-59-43-mozilla-beta/ 10-Dec-2012 15:51
  2012-12-10-15-03-46-mozilla-beta/ 10-Dec-2012 16:07
  2012-12-10-15-04-04-mozilla-beta/ 10-Dec-2012 17:06
  2012-12-10-15-04-06-mozilla-beta/ 10-Dec-2012 17:05
  2012-12-10-20-12-30-mozilla-beta/ 10-Dec-2012 20:28

Finding a build thus requires a more sophisticated algorithm than the one that mozdownload currently uses. It isn't enough to find all dirs for the given day/channel and then pick the last one. mozdownload must also filter the dirs by whether or not they contain an actual build.

Here's a branch that does that. Along the way, it fixes a recent regression that triggers an exception when checking if tmp_file is a file under certain circumstances in which tmp_file is undefined. And it enhances the directory parser to allow filtering by both regex and function.

@whimboo whimboo was assigned Dec 17, 2012
mykmelez added some commits Dec 17, 2012
@mykmelez mykmelez don't assume non-en-US build dirs have -l10n appended to their names …
…(as B2G's 'multi' localization build dirs do not)
dac7120
@mykmelez mykmelez Revert "even if a time has been included in a date for which to get b…
…uild info, don't assume a build necessarily exists at that time"

This reverts commit 9dd643d.
9f80994
@mykmelez mykmelez when retrieving the build status file for the latest build, make sure…
… we get the correct file for the locale
af8fa08
@mykmelez
Mozilla member

I found two more problems with the scraper code while trying to get the latest B2G build from the mozilla-b2g18 branch today.

The latest-mozilla-b2g18/ dir has two text files with info on Mac builds, b2g-18.0.en-US.mac64.txt, which points to the 2012-12-13-03-03-51-mozilla-b2g18/ directory; and b2g-18.0.multi.mac64.txt, which points to the 2012-12-13-09-53-15-mozilla-b2g18/ directory.

The first problem was that DailyScraper wouldn't necessarily get the right build status file when retrieving the latest build, since it didn't take the locale into account when retrieving the file; and later that would cause problems when it tried to download the build, since then it does take the locale into account.

I first fixed this by ignoring a mismatch between the date/time in the status file and the date/time of the build dir. But then I realized that the mismatch masked the real problem and reverted that change in favor of making sure DailyScraper retrieves the right build status file for the given locale.

The second problem was that get_build_info_for_date() assumes that build dirs for non-en-US builds have -l10n appended to their names, but the ones for B2G don't have that suffix. So I fixed this by making the suffix optional in the regex that identifies build dirs.

@mykmelez
Mozilla member

@whimboo: The last commit reverts a change you made in pull 31 that broke downloading of Windows B2G builds, which don't have an installer variant. See for example this directory containing the most recent Windows build:

https://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2012/12/2012-12-13-09-53-19-mozilla-b2g18/

Presumably you made the change for a reason, though, and I don't want to break that other use case! Perhaps you can elaborate, and then we can figure out how to keep both working?

@whimboo

@mykmelez, can you please put the last commit into a separate pull? It's not really related to that one and also a regression. Sorry for that, we need tests FWIW.

Not sure when I will find the time to check this pull request. I'm on PTO starting tomorrow. How critical is that fix for you?

@mykmelez
Mozilla member

@whimboo, no problem, I'll put the last commit into a separate pull.

And this fix isn't critical, as we can use my fork of mozdownload in the meantime. So don't worry about checking it today; let's pick it back up when you return from PTO!

@davehunt
Mozilla member

I've rebased and updated the changes from this pull and opened a new pull request as #99

@davehunt davehunt closed this Jul 16, 2013
@whimboo whimboo was unassigned by moz-hwine Aug 14, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment