Hyperband band indexes are off by one

[leopd]

wandb --version && python --version && uname

• Weights and Biases version: 0.9.7
• Python version: 3.7.9
• Operating System: Ubuntu 18.04LTS

Description

Using the hyperband early termination criterion for sweeps, you must specify where the "bands" are, which is how many iterations of the objective metric get reported before the algorithm considers whether or not to stop the run. They are an exponential series like 2,4,8,16, etc.

What I Did

If you configure the bands like:

early_terminate:
    type: hyperband
    min_iter: 2
    eta: 2

Here, the bands should be configured for iterations 2, 4, 8, 16, etc. But instead of considering the 2nd, 4th, 8th, 16th metric reported, it considers the 3rd, 5th, 9th, 17th, etc. This is because the code takes the human-specified iteration numbers which are 1-based (e.g. "1" means the first iteration) and uses them as indexes into a python list, which is 0-based. code

[issue-label-bot]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.73. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[cvphelps]

Hi @leopd thanks for reporting this off by one error, good catch. I'll track this internally.

[alexmirrington]

Hi @cvphelps, any ideas on the schedule for this bug and whether sprcifying the max_iter field in the sweep config also yields incorrect results?

[cvphelps]

Hi @alexmirrington thanks for your message. We will be including this in our first fixes for sweeps features after we finish the Client improvements this month. We'll be moving the client code from the client-ng repo in the next week or so.

For max_iter: if you've logged a metric 3 times it's a max_iter of 3 — there's a separation of iterations and steps, where steps start at 0 and iterations start at 1. This sounds like something we can clarify in the documentation.

[alexmirrington]

@cvphelps Cheers for the explanation :) For now, I'll just log a validation step at the start of the run to account for the index offset until the issue is fixed :)

[github-actions]

This issue is stale because it has been open 60 days with no activity.

[sydholl]

In the past year we've majorly reworked the CLI and UI for Weights & Biases. We're closing issues older than 6 months. Please comment to reopen.