172.17.0.1 - - [18/Jan/2022 01:08:06] "POST /inpaint HTTP/1.1" 200 -

Origin image shape: (3840, 2880, 3)

Resized image shape: (3840, 2880, 3)

[2022-01-18 01:10:11,646] ERROR in app: Exception on /inpaint [POST]

Traceback (most recent call last):

  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app

    response = self.full_dispatch_request()

  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request

    rv = self.handle_user_exception(e)

  File "/usr/local/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function

    return cors_after_request(app.make_response(f(*args, **kwargs)))

  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request

    rv = self.dispatch_request()

  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request

    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)

  File "main.py", line 63, in process

    res_np_img = run(image, mask)

  File "main.py", line 101, in run

    inpainted_image = model(image, mask)

  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl

    return forward_call(*input, **kwargs)

RuntimeError: The following operation failed in the TorchScript interpreter.

Traceback of TorchScript, serialized code (most recent call last):

  File "code/__torch__/saicinpainting/training/trainers/default.py", line 13, in forward

    masked_img = torch.mul(img, torch.rsub(mask, 1, 1))

    input = torch.cat([masked_img, mask], 1)

    _1 = torch.mul(mask, (_0).forward(input, ))

                          ~~~~~~~~~~~ <--- HERE

    _2 = torch.mul(torch.rsub(mask, 1, 1), img)

    return torch.add(_1, _2, alpha=1)

  File "code/__torch__/saicinpainting/training/modules/ffc.py", line 9, in forward

  def forward(self: __torch__.saicinpainting.training.modules.ffc.FFCResNetGenerator,

    input: Tensor) -> Tensor:

    return (self.model).forward(input, )

            ~~~~~~~~~~~~~~~~~~~ <--- HERE

class FFC_BN_ACT(Module):

  __parameters__ = []

  File "code/__torch__/torch/nn/modules/container/___torch_mangle_818.py", line 78, in forward

    _32 = getattr(self, "1")

    _33 = (getattr(self, "0")).forward(input, )

    _34 = (_31).forward((_32).forward(_33, ), )

                         ~~~~~~~~~~~~ <--- HERE

    _35 = (_29).forward((_30).forward(_34, ), )

    _36, _37, = _35

  File "code/__torch__/saicinpainting/training/modules/ffc.py", line 25, in forward

    _1 = self.bn_g

    _2 = self.act_l

    _3 = (self.bn_l).forward((self.ffc).forward(argument_1, ), )

          ~~~~~~~~~~~~~~~~~~ <--- HERE

    _4 = (_2).forward(_3, )

    _5 = (_1).forward()

  File "code/__torch__/torch/nn/modules/batchnorm.py", line 16, in forward

    _1 = self.running_mean

    _2 = self.bias

    input = torch.batch_norm(argument_1, self.weight, _2, _1, _0, False, 0.10000000000000001, 1.0000000000000001e-05, True)

            ~~~~~~~~~~~~~~~~ <--- HERE

    return input


Traceback of TorchScript, original code (most recent call last):

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/functional.py(2150): batch_norm

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py(140): forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl

/root/home/code/lama/saicinpainting/training/modules/ffc.py(253): forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/container.py(119): forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl

/root/home/code/lama/saicinpainting/training/modules/ffc.py(367): forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl

/root/home/code/lama/saicinpainting/training/trainers/default.py(83): forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/jit/_trace.py(940): trace_module

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/torch/jit/_trace.py(742): trace

bin/to_jit.py(46): main

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/core/utils.py(160): run_job

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/hydra.py(105): run

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/utils.py(371): <lambda>

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/utils.py(211): run_and_report

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/_internal/utils.py(368): _run_hydra

/opt/venv/ocr-detection-detectron2/lib/python3.6/site-packages/hydra/main.py(53): decorated_main

bin/to_jit.py(59): <module>

RuntimeError: CUDA error: unknown error