How to save and load the fitted model architecture?

[zym604]

Hi,

Thank you for providing such wonderful work!

My question may be simple: I use the fit function to get the best model architecture and model parameters, optimizer, etc. But how can I save the result and load it in another file?

For example, now I can see the model's architecture by:

>>> model = autonet.get_pytorch_model()
>>> model.eval()
Out[45]: 
Sequential(
  (0): Linear(in_features=2, out_features=437, bias=True)
  (1): Sequential(
    (0): ResBlock(
      (shortcut): Linear(in_features=437, out_features=147, bias=True)
      (start_norm): Sequential(
        (0): BatchNorm1d(437, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU()
      )
      (layers): Sequential(
        (0): Linear(in_features=437, out_features=147, bias=True)
        (1): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU()
        (3): Dropout(p=0.30851922736236553)
        (4): Linear(in_features=147, out_features=147, bias=True)
      )
    )
    (1): ResBlock(
      (layers): Sequential(
        (0): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU()
        (2): Linear(in_features=147, out_features=147, bias=True)
        (3): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): ReLU()
        (5): Dropout(p=0.30851922736236553)
        (6): Linear(in_features=147, out_features=147, bias=True)
      )
    )
  )
  (2): Sequential(
    (0): ResBlock(
      (layers): Sequential(
        (0): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU()
        (2): Linear(in_features=147, out_features=147, bias=True)
        (3): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): ReLU()
        (5): Dropout(p=0.0)
        (6): Linear(in_features=147, out_features=147, bias=True)
      )
    )
    (1): ResBlock(
      (layers): Sequential(
        (0): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU()
        (2): Linear(in_features=147, out_features=147, bias=True)
        (3): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): ReLU()
        (5): Dropout(p=0.0)
        (6): Linear(in_features=147, out_features=147, bias=True)
      )
    )
  )
  (3): BatchNorm1d(147, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (4): ReLU()
  (5): Linear(in_features=147, out_features=1, bias=True)
)

But how can I reuse the structure in another file? Just manually typing? I think there must be some smart way to do this. May 'rifit' can do the job, I just don't know how.

However, refit might be useful if you want to fit on the full dataset or even another dataset or if you just want to fit a model without searching.

[zym604]

I have a stupid way of saving the model by:

torch.save(model,PATH)

But it only saves the model's architecture and weight/bias. It doesn't save optimizer information. How can I get the optimizer object? Anything like autonet.get_pytorch_optimizer()?

There must be some smart way to do it. I just don't know. Please help.

[LMZimmer]

Hey,

thank you for the kind words and the feedback. We will include a get_optimizer method soon, for now here is a workaround:

from autoPyTorch.pipeline.nodes.optimizer_selector import OptimizerSelector autonet.pipeline[OptimizerSelector.get_name()].fit_output["optimizer"]

If you use the 'refit', the autonet will automatically use the incumbent configuration obtained in the last 'fit' call.

[zym604]

Thank you very much, how can I tell the Autonet to use a fixed network architecture? like:

model_fix = autonet.get_pytorch_model()
res = autonet.fit(X_train=x_train, Y_train=y_train, X_valid=x_dev, Y_valid=y_dev, model = model_fix)

[zym604]

Hey,

thank you for the kind words and the feedback. We will include a get_optimizer method soon, for now here is a workaround:

from autoPyTorch.pipeline.nodes.optimizer_selector import OptimizerSelector autonet.pipeline[OptimizerSelector.get_name()].fit_output["optimizer"]

If you use the 'refit', the autonet will automatically use the incumbent configuration obtained in the last 'fit' call.

Thank you for your reply. Please have a look at my question about fix network architecture in refit function at your convenience.

[LMZimmer]

If you want to search on a specific space you could do, e.g.:

autonet.fit(X_train=x_train, Y_train=y_train, X_valid=x_dev, Y_valid=y_dev, networks=['resnet'])

If you want to just fit a specific configuration, you pass a dictionary to a refit call:

hyperparameter_config = {
...
    'NetworkSelectorDatasetInfo:darts:layers': 20,
    'CreateImageDataLoader:batch_size': 58,
    'ImageAugmentation:augment': True,
    'ImageAugmentation:cutout': True,
    'LossModuleSelectorIndices:loss_module': 'cross_entropy',
...
}

autonet.refit(X_train=X_train,
                    Y_train=Y_train,
                    X_valid=None,
                    Y_valid=None,
                    hyperparameter_config=hyperparameter_config,
                    autonet_config=autonet.get_current_autonet_config(),
                    budget=50)

I also recommend the jupyter notebook tutorial.

[zym604]

If you want to search on a specific space you could do, e.g.:

autonet.fit(X_train=x_train, Y_train=y_train, X_valid=x_dev, Y_valid=y_dev, networks=['resnet'])

If you want to just fit a specific configuration, you pass a dictionary to a refit call:

hyperparameter_config = {
...
    'NetworkSelectorDatasetInfo:darts:layers': 20,
    'CreateImageDataLoader:batch_size': 58,
    'ImageAugmentation:augment': True,
    'ImageAugmentation:cutout': True,
    'LossModuleSelectorIndices:loss_module': 'cross_entropy',
...
}

autonet.refit(X_train=X_train,
                    Y_train=Y_train,
                    X_valid=None,
                    Y_valid=None,
                    hyperparameter_config=hyperparameter_config,
                    autonet_config=autonet.get_current_autonet_config(),
                    budget=50)

I also recommend the jupyter notebook tutorial.

Thank you! That solves my problem. I'm actually trying to save and load the fit result using a file so that I could refit without the fit process. Now according to your answer I write the following code:

results_fit = autonet.fit(X_train=X_train,
                          Y_train=Y_train,
                          validation_split=0.3,
                          max_runtime=300,
                          min_budget=60,
                          max_budget=100,
                          refit=True)
# Save fit results as json
with open("logs/results_fit.json", "w") as file:
    json.dump(results_fit, file)
# Load jason data
with open("logs/results_fit.json") as file:
    results_fit = json.load(file)
# refit with the fitted result and more budget
results_refit50 = autonet.refit(X_train=X_train,
                              Y_train=Y_train,
                              X_valid=None,
                              Y_valid=None,
                              hyperparameter_config=results_fit['optimized_hyperparameter_config'],
                              autonet_config=autonet.get_current_autonet_config(),
                              budget=50)
results_refit500 = autonet.refit(X_train=X_train,
                              Y_train=Y_train,
                              X_valid=None,
                              Y_valid=None,
                              hyperparameter_config=results_fit['optimized_hyperparameter_config'],
                              autonet_config=autonet.get_current_autonet_config(),
                              budget=500)
results_refit5000 = autonet.refit(X_train=X_train,
                              Y_train=Y_train,
                              X_valid=None,
                              Y_valid=None,
                              hyperparameter_config=results_fit['optimized_hyperparameter_config'],
                              autonet_config=autonet.get_current_autonet_config(),
                              budget=5000)

According to the result, training accuracy do increase with budget:

>>> results_refit50['info']['train_accuracy']
49.2436974789916
>>> results_refit500['info']['train_accuracy']
73.94285714285715
>>> results_refit5000['info']['train_accuracy']
100.0

[LMZimmer]

I am glad I could help. You can also set use_tensorboard_logger=True to see the learning curves with tensorboard.