Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

transformer aren't generate #224

Open
Christophe-pere opened this issue Feb 19, 2020 · 2 comments
Open

transformer aren't generate #224

Christophe-pere opened this issue Feb 19, 2020 · 2 comments

Comments

@Christophe-pere
Copy link

Christophe-pere commented Feb 19, 2020 •
edited

Hi there,

When I launch the command :

python3 main.py train --pipeline_name unet_weighted

I have this issue :

2020-02-19 00-01-08 steps >>> step category_mapper transforming...

100%|##########| 1000/1000 [00:00<00:00, 1016.60it/s]2020-02-19 00-01-09 steps >>> step labeler adapting inputs
2020-02-19 00-01-09 steps >>> step labeler transforming...

100%|##########| 1000/1000 [00:01<00:00, 542.57it/s]2020-02-19 00-01-10 steps >>> step mask_resize adapting inputs
2020-02-19 00-01-10 steps >>> step mask_resize transforming...

100%|##########| 1000/1000 [00:18<00:00, 54.03it/s]2020-02-19 00-01-29 steps >>> step score_builder adapting inputs
2020-02-19 00-01-29 steps >>> step score_builder transforming...

100%|##########| 1000/1000 [00:09<00:00, 110.19it/s]2020-02-19 00-01-38 steps >>> step output adapting inputs
2020-02-19 00-01-38 steps >>> step output transforming...
2020-02-19 00-01-38 steps >>> Creating annotations
2020-02-19 00-01-51 steps >>> Calculating mean precision and recall
loading annotations into memory...
Done (t=5.64s)
creating index...
index created!
Loading and preparing results...
DONE (t=0.64s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *segm*
DONE (t=5.08s).
Accumulating evaluation results...
DONE (t=0.21s).
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.727
 Average Precision  (AP) @[ IoU=0.50      | area= small | maxDets=100 ] = 0.159
 Average Precision  (AP) @[ IoU=0.50      | area= large | maxDets=100 ] = 0.817
 Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.804
 Average Recall     (AR) @[ IoU=0.50      | area= small | maxDets=100 ] = 0.386
 Average Recall     (AR) @[ IoU=0.50      | area= large | maxDets=100 ] = 0.882
2020-02-19 00-02-04 steps >>> epoch 0 validation sum:     0.72717
2020-02-19 00-02-04 steps >>> epoch 0 model saved to data/experiments/mapping_challenge_baseline/checkpoints/unet/best.torch
2020-02-19 00-02-04 steps >>> epoch 1 current lr: 0.0005

2020-02-19 00-02-06 steps >>> training finished
Traceback (most recent call last):
  File "main.py", line 68, in <module>
    main()
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "main.py", line 30, in train
    pipeline_manager.train(pipeline_name, dev_mode)
  File "/home/open-solution-mapping-challenge/src/pipeline_manager.py", line 42, in train
    train(pipeline_name, dev_mode, self.logger, self.params, self.seed)
  File "/home/open-solution-mapping-challenge/src/pipeline_manager.py", line 136, in train
    pipeline.fit_transform(data)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 106, in fit_transform
    step_inputs[input_step.name] = input_step.fit_transform(data)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 106, in fit_transform
    step_inputs[input_step.name] = input_step.fit_transform(data)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 106, in fit_transform
    step_inputs[input_step.name] = input_step.fit_transform(data)
  [Previous line repeated 4 more times]
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 112, in fit_transform
    return self._cached_fit_transform(step_inputs)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 123, in _cached_fit_transform
    step_output_data = self.transformer.fit_transform(**step_inputs)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 263, in fit_transform
    return self.transform(*args, **kwargs)
  File "/home/open-solution-mapping-challenge/src/models.py", line 89, in transform
    outputs = self._transform(datagen, validation_datagen)
  File "/home/open-solution-mapping-challenge/src/steps/pytorch/models.py", line 131, in _transform
    outputs_batch = self.model(X)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 71, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/open-solution-mapping-challenge/src/unet_models.py", line 387, in forward
    conv2 = self.conv2(conv1)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torchvision/models/resnet.py", line 81, in forward
    out = self.bn2(out)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 37, in forward
    self.training, self.momentum, self.eps)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/functional.py", line 1013, in batch_norm
    return f(input, weight, bias)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 175, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 28632) is killed by signal: Killed.

I tried the code on only one epoch to see how it works.

The code is running on a VM Ubuntu 18.04 with 2 GPUs.

How can I correct the code to generate the transformers layer ?

Best,

Chris
@Christophe-pere Christophe-pere changed the title ValueError: No transformer cached unet transformer aren't generate Feb 19, 2020
@Christophe-pere
Copy link
Author

After following the advice of the issue #210 the evaluation and prediction are right. But, I tried after that to train again the model of the first issue bye overwriting the previous model as #160 explained. The training is gonna be right except finishing with the same error as previously shown.
But when I ran the evaluation I got this issue :

2020-02-19 14-02-30 mapping-challenge >>> evaluating
2020-02-19 14-02-37 steps >>> step xy_inference adapting inputs
2020-02-19 14-02-37 steps >>> step xy_inference transforming...
2020-02-19 14-02-37 steps >>> step xy_inference adapting inputs
2020-02-19 14-02-37 steps >>> step xy_inference transforming...
2020-02-19 14-02-37 steps >>> step loader adapting inputs
2020-02-19 14-02-37 steps >>> step loader transforming...
2020-02-19 14-02-37 steps >>> step unet unpacking inputs
2020-02-19 14-02-37 steps >>> step unet loading transformer...
Traceback (most recent call last):
  File "main.py", line 68, in <module>
    main()
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "main.py", line 41, in evaluate
    pipeline_manager.evaluate(pipeline_name, dev_mode, chunk_size)
  File "/home/open-solution-mapping-challenge/src/pipeline_manager.py", line 52, in evaluate
    evaluate(pipeline_name, dev_mode, chunk_size, self.logger, self.params, self.seed)
  File "/home/open-solution-mapping-challenge/src/pipeline_manager.py", line 152, in evaluate
    prediction = generate_prediction(meta_valid, pipeline, logger, CATEGORY_IDS, chunk_size, params.num_threads)
  File "/home/open-solution-mapping-challenge/src/pipeline_manager.py", line 190, in generate_prediction
    return _generate_prediction(meta_data, pipeline, logger, category_ids, num_threads)
  File "/home/open-solution-mapping-challenge/src/pipeline_manager.py", line 203, in _generate_prediction
    output = pipeline.transform(data)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 158, in transform
    step_inputs[input_step.name] = input_step.transform(data)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 158, in transform
    step_inputs[input_step.name] = input_step.transform(data)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 158, in transform
    step_inputs[input_step.name] = input_step.transform(data)
  [Previous line repeated 4 more times]
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 164, in transform
    return self._cached_transform(step_inputs)
  File "/home/open-solution-mapping-challenge/src/steps/base.py", line 170, in _cached_transform
    self.transformer.load(self.cache_filepath_step_transformer)
  File "/home/open-solution-mapping-challenge/src/steps/pytorch/models.py", line 156, in load
    self.model.load_state_dict(torch.load(filepath))
  File "/home/open-solution-mapping-challenge/mapping/lib/python3.6/site-packages/torch/nn/modules/module.py", line 522, in load_state_dict
    .format(name))
KeyError: 'unexpected key "module.module.encoder.conv1.weight" in state_dict'

How can correct this ?

Best,

Chris

@jakubczakon
Copy link
Collaborator

Hi @Christophe-pere and apologies for taking this long to answer (was unwatched from this repo for some reason).

The loading issue is some pickling problem connected to running it on multigpu (not 100% sure but likely).

You can fix it by overriding how the model gets loaded in here:
https://github.com/neptune-ai/open-solution-mapping-challenge/blob/master/src/models.py

   def fit(self, datagen, validation_datagen=None, meta_valid=None):
        self._initialize_model_weights()

        self.model = nn.DataParallel(self.model)

I hope this helps!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jakubczakon @Christophe-pere