How to set max_compositions

[hlzhang109]

Hi Chen, do you know how to set the max_compositions/steps param? The default at

private-transformers/private_transformers/privacy_utils/privacy_engine.py

Line 148 in 684e27f

self.steps = 0 # Tracks privacy spending.

is 0 but would raise an error

private-transformers/private_transformers/privacy_utils/accounting/gdp_accounting.py:33: RuntimeWarning: invalid value encountered in double_scalars return norm.cdf(-eps / mu + mu / 2) - np.exp(eps) * norm.cdf(-eps / mu - mu / 2)
rv_accountant/accountant.py:55: RuntimeWarning: divide by zero encountered in double_scalars mesh_size = 2*eps_error / np.sqrt(2*max_compositions*np.log(2/eta0))

[lxuechen]

Great point. Ideally, we'd only call the privacy accounting function after we've done a gradient update, in which case we'd get strictly positive self.steps (note self.steps is updated in the step function).

My code runs a prelim. eval. before any parameter update just to check the initial performance model of the model. This produces a warning you'd get, since the privacy accounting code is also ran. This doesn't really affect the correctness however.

[lxuechen]

By the way, self.steps is not the max composition steps, but rather the total number of updates already performed by the optimizer.

[hlzhang109]

Thanks Chen for the explanation! I set the dataset size too small when debugging so self.steps didn't get updated as gradient_accumulation_steps was never reached.