Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import ONNX model from pytorch - LDense only works over 2D tensors (LDense) #340

Closed
FlorianThaler opened this issue Oct 2, 2022 · 6 comments

Comments

@FlorianThaler
Copy link

FlorianThaler commented Oct 2, 2022

Hello.

I trained a model (using dense layers only) in stable-baselines, which I would like to include into an EDDL project. For this reason I imported the model in pytorch, which allows me to export the model in onnx format. When importing and building the onnx model in eddl I encounter the following error:

    ==================================================================
    ⚠️  LDense only works over 2D tensors (LDense) ⚠️
    ==================================================================
    
    terminate called after throwing an instance of 'std::runtime_error'
      what():  RuntimeError: LDense
    Aborted (core dumped)

For the sake of reproducibility I post the codes of a dummy example down below.

  1. The definition of the stable-baselines model and the conversion to pytorch, and onnx (versions: onnx == 1.9.0, torch == 1.10.1, stable-baselines == 2.10.0):

       ```
       import torch as th
       import torch.nn as nn
       
       import gym
       
       from stable_baselines import PPO1
       from stable_baselines.common.policies import MlpPolicy
       
       class PyTorchMlp(nn.Module):  
           def __init__(self, n_inputs=4, n_actions=2):
               nn.Module.__init__(self)
       
               self.fc1 = nn.Linear(n_inputs, 64)
               self.fc2 = nn.Linear(64, 64)      
               self.fc3 = nn.Linear(64, n_actions)      
               self.activ_fn = nn.Tanh()
               self.out_activ = nn.Softmax(dim=0)
       
           def forward(self, x):
               x = self.activ_fn(self.fc1(x))
               x = self.activ_fn(self.fc2(x))
               x = self.out_activ(self.fc3(x))
               return x
       
       def copy_mlp_weights(baselines_model):
           torch_mlp = PyTorchMlp(n_inputs=4, n_actions=2)
           model_params = baselines_model.get_parameters()
           
           policy_keys = [key for key in model_params.keys() if "pi" in key]
           policy_params = [model_params[key] for key in policy_keys]
               
           for (th_key, pytorch_param), key, policy_param in zip(torch_mlp.named_parameters(), policy_keys, policy_params):
               param = th.from_numpy(policy_param)    
               #Copies parameters from baselines model to pytorch model
               print(th_key, key)
               print(pytorch_param.shape, param.shape, policy_param.shape)
               pytorch_param.data.copy_(param.data.clone().t())
               
           return torch_mlp
       
       
       env = gym.make("CartPole-v1")
       obs = env.reset()   
       
       
       model = PPO1(MlpPolicy, env, verbose = 0)
       model.learn(total_timesteps = 25)
       
       for key, value in model.get_parameters().items():
           print(key, value.shape)
       
       th_model = copy_mlp_weights(model)
       
       dummy_input = th.randn(4)
       
       th.onnx.export(th_model, dummy_input, 'test_0.onnx', opset_version = 9, export_params = True, verbose = True,\
                           input_names = ['input'], output_names = ['output'])
       ```
    
  2. The C++ integration:

          #include <iostream>
          
          #include "eddl/apis/eddl.h"
          #include "eddl/serialization/onnx/eddl_onnx.h"
          
          using namespace eddl;
          
          int main(int argc, char **argv)
          {
          
              // import model
              Net* net = import_net_from_onnx_file("../data/test_0.onnx");
              // Net* net = import_net_from_onnx_file("../data/model.onnx");
              // build model
              build(net);
              // build(net,
              //     adam(0.01),                 // optimizer
              //     {"mean_squared_error"},     // losses
              //     {"categorical_accuracy"},   // metrics
              //     CS_CPU(),                   // CPU with maximum threads availables
              //     false);                     // no initialisation of weights
              Tensor* inpt = Tensor::randu({1, 4});
              summary(net);
          
              // auto outpt = predict(net, {inpt});
          }

As I found, a similar issue was already discussed in #328.

@salvacarrion
Copy link
Contributor

salvacarrion commented Oct 3, 2022

Which version of the EDDL are you using? In theory, we support n-dimensional dense layers since the last version 1.1b, which is on par with the latest PyEDDL version (1.3.*)

@FlorianThaler
Copy link
Author

I pulled the most recent version from the main branch.

The input the neural network takes is no multi-dimensional tensor - it is only an array with four floats. Is there Is there another way to import a neural network? Would it be feasible to import the weights of a neural network using the function eddl::set_parameters? Is there a detailed description of its functionality in the docu?

@salvacarrion
Copy link
Contributor

If you are using the last version, the n-dense should work.
We only support onnx for IO, although you could indeed try to load the weights "manually" using set_parameters.

Anyway, could you send me the onnx file to take a look and debug it?

@FlorianThaler
Copy link
Author

onnx_model.zip

Sure - it would be great if you could have a look at it. It seems I can't add the file directly - so I compress it into a zip file. Hope it works ...

@jonandergomez
Copy link
Contributor

Dear Florian,

Salva and myself have been working on the code of the EDDL to find the cause of the error.

The error is generated when importing from ONNX if the shape of the input layer(s) has only one dimension. The EDDL always assumes the first component of the shape is the batch_size. We corrected the problem with the C++ version with a provisional patch for checking whether the shape of input layers has only one dimension, then using it as the size of the input layer.

At least for the purpose of UC7 from the FRACTAL project, it runs OK now only in C++.

@FlorianThaler
Copy link
Author

FlorianThaler commented Oct 12, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants