[CLI]: When resuming run with high-precision hyperparam wandb.config.update() throws an Exception

[Niccolo-Ajroldi]

Describe the bug

When resuming a run, wandb.config.update throws an Exception when using hyperparameters with high level of precision.

Minimal Example

Execute this code a first time to initialize a run.
Execute it a second time to simulate resuming of the same run. Here an exception occurs.

import wandb

dir = "/Users/nico/code_exp/exp1"
hyperparameters = {"LR": 0.00018805814126031094,}

wandb.init(dir=dir, resume='allow', id="id_01")
wandb.config.update(hyperparameters) # this throws an exception the second time the script is executed

Traceback of the exception:

wandb: ERROR Attempted to change value of key "LR" from 0.00018805814126031096 to 0.00018805814126031094
wandb: ERROR If you really want to do this, pass allow_val_change=True to config.update()
Traceback (most recent call last):
  File "wandb_err.py", line 12, in <module>
    wandb.config.update(hyperparameters)
  File "/Users/nico/miniconda3/envs/wandb_err/lib/python3.8/site-packages/wandb/sdk/wandb_config.py", line 184, in update
    sanitized = self._update(d, allow_val_change)
  File "/Users/nico/miniconda3/envs/wandb_err/lib/python3.8/site-packages/wandb/sdk/wandb_config.py", line 177, in _update
    sanitized = self._sanitize_dict(
  File "/Users/nico/miniconda3/envs/wandb_err/lib/python3.8/site-packages/wandb/sdk/wandb_config.py", line 264, in _sanitize_dict
    k, v = self._sanitize(k, v, allow_val_change)
  File "/Users/nico/miniconda3/envs/wandb_err/lib/python3.8/site-packages/wandb/sdk/wandb_config.py", line 285, in _sanitize
    raise config_util.ConfigError(
wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "LR" from 0.00018805814126031096 to 0.00018805814126031094
If you really want to do this, pass allow_val_change=True to config.update()

Where does the value 0.00018805814126031096 come from? It seems that wandb.init does not read correctly the hyperparameters in the config file.

Environment

WandB version: 0.16.3

OS: macOS Sonoma 14.2, but also on Ubuntu 22.04.3

Python version: 3.8.18

[thanos-wandb]

Hi @Niccolo-Ajroldi thank you for reporting this. It seems from the stack trace that the value 0.00018805814126031096 is what the run had in the config prior to resuming to this run. Regarding the error, the Sweep config is protected therefore you could update it if you explicitly pass the boolean arg as follows:
wandb.config.update(hyperparameters, allow_val_change=True)

Would this work for you?

[Niccolo-Ajroldi]

Hi @thanos-wandb, thank you for looking at this.

Please look better at my example: the saved value shoul be 0.00018805814126031094, but when resuming, this seems to magically be 0.00018805814126031096, hence the Exception.

If you just run my minimal example twice (1 time to mimic the original run, and a second time to mimic resuming), you will face the same Exception.

[thanos-wandb]

Hi @Niccolo-Ajroldi thank you for clarifying this. Looking further what's causing this discrepancy and I will get back to you with more information.