respect time_limits better

[jwmueller]

For num_trials default:

if time_limits < THRESHOLD:
num_trials = 1

where THRESHOLD is something small, say = 1 or 10

[zhanghang1989]

Maybe we can do this

num_trials = max(get_cpu_count() // num_cpus, get_cpu_count() // num_gpus) if time_limits < THRESHOLD and num_gpus > 0 else num_trials

[jwmueller]

I think we should try to primarily control everything based on time_limits (so assume most users do not specify num_trials). Something like this:

def task.fit(..., time_limits = None, num_trials=None):
   if num_trials is None: 
        if time_limits is None:
            time_limits = 10 * 60  # run for 10min by default
        elif time_limits <= THRESHOLD: # THRESHOLD = say 1 or 10 sec, too little time to do much
            num_trials = 1
       else:
            num_trials = np.inf  # run as many trials as you can within the given time_limits
   if time_limits is None:
      time_limits = np.inf # in this case, user only specified num_trials, so just run all of them regardless of time-limits

[jwmueller]

Currently the user has to explicitly be aware of both time_limits and num_trials. When I don't specify num_trials, it seems to default to 2 beneath the hood. This is very confusing for me, since when I set time_limits = big number, I expect task.fit() to run lots of trials. Currently task.fit() is basically running for min{ num_trials, time_limits} which is not very intuitive behavior. I think the default behavior outlined in my code above would be much more natural, where whichever of num_trials or time_limits is specified is the one that dictates the behavior of task.fit()

[cgraywang]

Why not leveraging soft time limits as previously implemented?

[jwmueller]

can you explain soft time limits?
My point is that most users will not be aware of both time_limits and num_trials, and they should not have to specify both of these things to control the overall runtime of autogluon. When they only specify one of them, the other one still heavily affects runtimes underneath the hood, which may be very confusing for users (it was for me)

[cgraywang]

Ok, I agree that the users only need to specify what they want and the remaining should stay as the default. Soft time limit is different, it means that for the trials already scheduled, we let them run to the end, similar to what we have.

[Innixma]

Resolved in current version