Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Post processing errors putting worker into maintenance #25

Closed
jug-dev opened this issue Mar 22, 2023 · 2 comments
Closed

Post processing errors putting worker into maintenance #25

jug-dev opened this issue Mar 22, 2023 · 2 comments

Comments

@jug-dev
Copy link
Contributor

jug-dev commented Mar 22, 2023

38 of same error about realESRGAN. Some of the stack trace reproduced below.

Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[124, 100,  64],
                │    │                                [128,  94,  45],
                │    │                                [152, 116,  65],
                │    │                                ...,
                │    │                                [176, 166, 149],
                │    │                                [179...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x000002188703D240>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x0000021887018700>, array([[[124, 100,  64],
           │               [128,  94,  45],
           │               [152...
           └ <function RealESRGANer.enhance at 0x0000021886E42560>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x0000021886E42440>
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2088
    │    │            └ 2048
    │    └ tensor([[[[ 0.2537,  0.1562,  0.0859,  ...,  0.5908,  0.6167,  0.5981],
    │                [ 0.1807,  0.1272,  0.0630,  ...,  0.5913, ...
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

RuntimeError: The expanded size of the tensor (0) must match the existing size (40) at non-singleton dimension 3.  Target sizes: [1, 3, 0, 0].  Tensor sizes: [3, 40, 40]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x000002188F100D30>
    └ <Thread(ThreadPoolExecutor-0_2, started 14092)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x000002188F100A60>
    └ <Thread(ThreadPoolExecutor-0_2, started 14092)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_2, started 14092)>
    │    │        │    └ (<weakref at 0x0000021B27C41210; to 'ThreadPoolExecutor' at 0x00000218B8504C10>, <_queue.SimpleQueue object at 0x0000021886A8...
    │    │        └ <Thread(ThreadPoolExecutor-0_2, started 14092)>
    │    └ <function _worker at 0x00000218B6591A20>
    └ <Thread(ThreadPoolExecutor-0_2, started 14092)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x00000218B6591B40>
    └ <concurrent.futures.thread._WorkItem object at 0x0000021ABFCAFE50>

> File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000021ABFCAFE50>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000021ABFCAFE50>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021B...
             └ <concurrent.futures.thread._WorkItem object at 0x0000021ABFCAFE50>

  File "F:\AI\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 423, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x00000218B9217DC0>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021BC524EBC0>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21AB0331690>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021BC524EBC0>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000218836D8430>
    │    └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21AB0331690>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021BC524EBC0>

  File "F:\AI\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21AB0331690>
    └ <nataili.esrgan.esrgan.esrgan object at 0x000002188703D240>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[124, 100,  64],
                   │    │       │              [128,  94,  45],
                   │    │       │              [152, 116,  65],
                   │    │       │              ...,
                   │    │       │              [176, 166, 149],
                   │    │       │              [179...
                   │    │       └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21AB0331690>
                   │    └ <function esrgan.process at 0x00000218836D9BD0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x000002188703D240>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21AB0331690>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x000002188703D240>
                   │    └ <function esrgan.esrgan_upscale at 0x00000218836D9C60>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x000002188703D240>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21AB0331D20>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x0000021BC518CA60>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.2510, 0.1765, 0.2549,  ..., 0.5843, 0.5961, 0.5686],
             │                 [0.1804, 0.1294, 0.2235,  ..., 0.5922, 0.5882, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:23:52.812599 | worker.workers.framework:check_running_job_status:132 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:23:52.909916 | worker.workers.framework:check_running_job_status:133 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 01:24:13.418065 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (2276), thread 'ThreadPoolExecutor-0_0' (10876):
Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[226,  46,  40],
                │    │                                [248,  35,  21],
                │    │                                [252,  28,  23],
                │    │                                ...,
                │    │                                [ 45, 229, 255],
                │    │                                [ 44...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x00000218F3078820>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BC524E7D0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x00000218F3078820>, array([[[226,  46,  40],
           │               [248,  35,  21],
           │               [252...
           └ <function RealESRGANer.enhance at 0x0000021886E42560>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x0000021886E42440>
    └ <realesrgan.utils.RealESRGANer object at 0x00000218F3078820>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 3112
    │    │            └ 2048
    │    └ tensor([[[[ 1.0657e-01,  6.1035e-02,  4.7974e-02,  ...,  1.0049e+00,
    │                  1.0059e+00,  1.0068e+00],
    │                [ 1.4075...
    └ <realesrgan.utils.RealESRGANer object at 0x00000218F3078820>

RuntimeError: The expanded size of the tensor (0) must match the existing size (40) at non-singleton dimension 3.  Target sizes: [1, 3, 1024, 0].  Tensor sizes: [3, 1064, 40]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x000002188F100D30>
    └ <Thread(ThreadPoolExecutor-0_0, started 10876)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x000002188F100A60>
    └ <Thread(ThreadPoolExecutor-0_0, started 10876)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_0, started 10876)>
    │    │        │    └ (<weakref at 0x00000218F30CBFB0; to 'ThreadPoolExecutor' at 0x00000218B8504C10>, <_queue.SimpleQueue object at 0x0000021886A8...
    │    │        └ <Thread(ThreadPoolExecutor-0_0, started 10876)>
    │    └ <function _worker at 0x00000218B6591A20>
    └ <Thread(ThreadPoolExecutor-0_0, started 10876)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x00000218B6591B40>
    └ <concurrent.futures.thread._WorkItem object at 0x0000021B3E6756C0>

> File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000021B3E6756C0>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000021B3E6756C0>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021B...
             └ <concurrent.futures.thread._WorkItem object at 0x0000021B3E6756C0>

  File "F:\AI\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 423, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x00000218B9217DC0>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021B3E676860>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=512x768 at 0x21D3E018D60>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021B3E676860>
    │    │       │            └ 'RealESRGAN_x4plus_anime_6B'
    │    │       └ <function post_process at 0x00000218836D8430>
    │    └ <PIL.Image.Image image mode=RGB size=512x768 at 0x21D3E018D60>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021B3E676860>

  File "F:\AI\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=512x768 at 0x21D3E018D60>
    └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BC524E7D0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[226,  46,  40],
                   │    │       │              [248,  35,  21],
                   │    │       │              [252,  28,  23],
                   │    │       │              ...,
                   │    │       │              [ 45, 229, 255],
                   │    │       │              [ 44...
                   │    │       └ <PIL.Image.Image image mode=RGB size=512x768 at 0x21D3E018D60>
                   │    └ <function esrgan.process at 0x00000218836D9BD0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BC524E7D0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=512x768 at 0x21D3E018D60>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x00000218F3078820>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BC524E7D0>
                   │    └ <function esrgan.esrgan_upscale at 0x00000218836D9C60>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BC524E7D0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21BC524ECB0>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x00000218F3078820>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x0000021A8F5148B0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.1569, 0.0824, 0.0902,  ..., 1.0000, 0.9882, 0.9843],
             │                 [0.1137, 0.0745, 0.0745,  ..., 1.0000, 1.0000, 1....
             └ <realesrgan.utils.RealESRGANer object at 0x00000218F3078820>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:24:14.650745 | worker.workers.framework:check_running_job_status:132 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:24:14.653738 | worker.workers.framework:check_running_job_status:133 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 01:25:29.597798 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (2276), thread 'ThreadPoolExecutor-0_2' (14092):
Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[251, 237, 229],
                │    │                                [255, 243, 237],
                │    │                                [255, 244, 239],
                │    │                                ...,
                │    │                                [255, 244, 237],
                │    │                                [255...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x0000021B31427FD0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x0000021887018700>, array([[[251, 237, 229],
           │               [255, 243, 237],
           │               [255...
           └ <function RealESRGANer.enhance at 0x0000021886E42560>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x0000021886E42440>
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2088
    │    │            └ 2048
    │    └ tensor([[[[0.9419, 0.9194, 0.9160,  ..., 0.9331, 0.9355, 0.9360],
    │                [0.9214, 0.9238, 0.9233,  ..., 0.9336, 0.9346, 0....
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

RuntimeError: The expanded size of the tensor (0) must match the existing size (40) at non-singleton dimension 2.  Target sizes: [1, 3, 0, 2048].  Tensor sizes: [3, 40, 2048]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x000002188F100D30>
    └ <Thread(ThreadPoolExecutor-0_2, started 14092)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x000002188F100A60>
    └ <Thread(ThreadPoolExecutor-0_2, started 14092)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_2, started 14092)>
    │    │        │    └ (<weakref at 0x0000021B27C41210; to 'ThreadPoolExecutor' at 0x00000218B8504C10>, <_queue.SimpleQueue object at 0x0000021886A8...
    │    │        └ <Thread(ThreadPoolExecutor-0_2, started 14092)>
    │    └ <function _worker at 0x00000218B6591A20>
    └ <Thread(ThreadPoolExecutor-0_2, started 14092)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x00000218B6591B40>
    └ <concurrent.futures.thread._WorkItem object at 0x0000021A8F0BEB30>

> File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000021A8F0BEB30>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000021A8F0BEB30>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000218...
             └ <concurrent.futures.thread._WorkItem object at 0x0000021A8F0BEB30>

  File "F:\AI\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 423, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.75
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x00000218B9217DC0>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021886F87B20>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21BDC3D85E0>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021886F87B20>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000218836D8430>
    │    └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21BDC3D85E0>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000021886F87B20>

  File "F:\AI\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.75
    │                          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21BDC3D85E0>
    └ <nataili.esrgan.esrgan.esrgan object at 0x0000021B31427FD0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.75}
                   │    │       │    └ array([[[251, 237, 229],
                   │    │       │              [255, 243, 237],
                   │    │       │              [255, 244, 239],
                   │    │       │              ...,
                   │    │       │              [255, 244, 237],
                   │    │       │              [255...
                   │    │       └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21BDC3D85E0>
                   │    └ <function esrgan.process at 0x00000218836D9BD0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000021B31427FD0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21BDC3D85E0>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x0000021B31427FD0>
                   │    └ <function esrgan.esrgan_upscale at 0x00000218836D9C60>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000021B31427FD0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21D1FFCEC50>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x0000021C44B2BBE0>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.8980, 0.9294, 0.9373,  ..., 0.9294, 0.9294, 0.9098],
             │                 [0.9216, 0.9412, 0.9451,  ..., 0.9373, 0.9333, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:25:31.570378 | worker.workers.framework:check_running_job_status:132 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:25:31.571377 | worker.workers.framework:check_running_job_status:133 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 01:26:23.858623 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (2276), thread 'ThreadPoolExecutor-0_1' (15960):
Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[136, 156, 181],
                │    │                                [133, 155, 184],
                │    │                                [132, 155, 182],
                │    │                                ...,
                │    │                                [243, 240, 239],
                │    │                                [242...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BA088C820>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x0000021887018700>, array([[[136, 156, 181],
           │               [133, 155, 184],
           │               [132...
           └ <function RealESRGANer.enhance at 0x0000021886E42560>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x0000021886E42440>
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 1320
    │    │            └ 0
    │    └ tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
    │                [0., 0., 0.,  ..., 0., 0., 0.],
    │                [0., 0., 0.,  ..., 0., 0., 0.],...
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

RuntimeError: The expanded size of the tensor (1280) must match the existing size (1320) at non-singleton dimension 2.  Target sizes: [1, 3, 1280, 2048].  Tensor sizes: [3, 1320, 2048]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x000002188F100D30>
    └ <Thread(ThreadPoolExecutor-0_1, started 15960)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x000002188F100A60>
    └ <Thread(ThreadPoolExecutor-0_1, started 15960)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_1, started 15960)>
    │    │        │    └ (<weakref at 0x00000218FB020DB0; to 'ThreadPoolExecutor' at 0x00000218B8504C10>, <_queue.SimpleQueue object at 0x0000021886A8...
    │    │        └ <Thread(ThreadPoolExecutor-0_1, started 15960)>
    │    └ <function _worker at 0x00000218B6591A20>
    └ <Thread(ThreadPoolExecutor-0_1, started 15960)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x00000218B6591B40>
    └ <concurrent.futures.thread._WorkItem object at 0x0000021D3E01B8B0>

> File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000021D3E01B8B0>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000021D3E01B8B0>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000218...
             └ <concurrent.futures.thread._WorkItem object at 0x0000021D3E01B8B0>

  File "F:\AI\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 423, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x00000218B9217DC0>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000218F3075FC0>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=1024x320 at 0x21B756E4BB0>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000218F3075FC0>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000218836D8430>
    │    └ <PIL.Image.Image image mode=RGB size=1024x320 at 0x21B756E4BB0>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000218F3075FC0>

  File "F:\AI\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=1024x320 at 0x21B756E4BB0>
    └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BA088C820>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[136, 156, 181],
                   │    │       │              [133, 155, 184],
                   │    │       │              [132, 155, 182],
                   │    │       │              ...,
                   │    │       │              [243, 240, 239],
                   │    │       │              [242...
                   │    │       └ <PIL.Image.Image image mode=RGB size=1024x320 at 0x21B756E4BB0>
                   │    └ <function esrgan.process at 0x00000218836D9BD0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BA088C820>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=1024x320 at 0x21B756E4BB0>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BA088C820>
                   │    └ <function esrgan.esrgan_upscale at 0x00000218836D9C60>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000021BA088C820>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x218870591B0>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x0000021B72A9BA30>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
             │                 [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:26:24.577326 | worker.workers.framework:check_running_job_status:132 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:26:24.579322 | worker.workers.framework:check_running_job_status:133 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 01:28:03.280712 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (2276), thread 'ThreadPoolExecutor-0_1' (15960):
Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[142,  99,   8],
                │    │                                [147, 103,   0],
                │    │                                [145, 104,   0],
                │    │                                ...,
                │    │                                [143, 102,   0],
                │    │                                [144...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x000002188175BD60>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x0000021887018700>, array([[[142,  99,   8],
           │               [147, 103,   0],
           │               [145...
           └ <function RealESRGANer.enhance at 0x0000021886E42560>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x0000021886E42440>
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2600
    │    │            └ 2048
    │    └ tensor([[[[ 0.0765,  0.0135,  0.0155,  ..., -0.0168, -0.0216, -0.0178],
    │                [ 0.0348,  0.0016, -0.0022,  ..., -0.0169, ...
    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

RuntimeError: The expanded size of the tensor (512) must match the existing size (552) at non-singleton dimension 3.  Target sizes: [1, 3, 512, 512].  Tensor sizes: [3, 552, 552]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x000002188F100D30>
    └ <Thread(ThreadPoolExecutor-0_1, started 15960)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x000002188F100A60>
    └ <Thread(ThreadPoolExecutor-0_1, started 15960)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_1, started 15960)>
    │    │        │    └ (<weakref at 0x00000218FB020DB0; to 'ThreadPoolExecutor' at 0x00000218B8504C10>, <_queue.SimpleQueue object at 0x0000021886A8...
    │    │        └ <Thread(ThreadPoolExecutor-0_1, started 15960)>
    │    └ <function _worker at 0x00000218B6591A20>
    └ <Thread(ThreadPoolExecutor-0_1, started 15960)>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x00000218B6591B40>
    └ <concurrent.futures.thread._WorkItem object at 0x0000021B27C32830>

> File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000021B27C32830>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000021B27C32830>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000224...
             └ <concurrent.futures.thread._WorkItem object at 0x0000021B27C32830>

  File "F:\AI\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 423, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x00000218B9217DC0>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000224202BEE60>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=640x640 at 0x21C5ED36350>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000224202BEE60>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000218836D8430>
    │    └ <PIL.Image.Image image mode=RGB size=640x640 at 0x21C5ED36350>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000224202BEE60>

  File "F:\AI\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=640x640 at 0x21C5ED36350>
    └ <nataili.esrgan.esrgan.esrgan object at 0x000002188175BD60>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[142,  99,   8],
                   │    │       │              [147, 103,   0],
                   │    │       │              [145, 104,   0],
                   │    │       │              ...,
                   │    │       │              [143, 102,   0],
                   │    │       │              [144...
                   │    │       └ <PIL.Image.Image image mode=RGB size=640x640 at 0x21C5ED36350>
                   │    └ <function esrgan.process at 0x00000218836D9BD0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x000002188175BD60>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=640x640 at 0x21C5ED36350>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x0000021887018700>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x000002188175BD60>
                   │    └ <function esrgan.esrgan_upscale at 0x00000218836D9C60>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x000002188175BD60>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x21A49501F60>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x0000021ABFF35480>

  File "F:\AI\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.0314, 0.0000, 0.0000,  ..., 0.1922, 0.1922, 0.1922],
             │                 [0.0000, 0.0000, 0.0000,  ..., 0.2000, 0.1961, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x0000021887018700>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:28:04.882257 | worker.workers.framework:check_running_job_status:132 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 01:28:04.883255 | worker.workers.framework:check_running_job_status:133 - 'RealESRGANer' object is not callable
NoneType: None
@donaldanixon
Copy link
Contributor

theres a double whammy there, for some reason the realesrgan itself has a tensor mismatch - which is weird, and then because that fails it tries to do the esrgan upscale, which doesn't work either

@jug-dev
Copy link
Contributor Author

jug-dev commented Mar 22, 2023

More stack traces:

ERROR      | 2023-03-22 22:29:17.872053 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (17536), thread 'ThreadPoolExecutor-0_2' (18840):
Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[168, 122,  78],
                │    │                                [173, 122,  62],
                │    │                                [170, 119,  61],
                │    │                                ...,
                │    │                                [162, 113,  59],
                │    │                                [162...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x00000245C2123310>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, array([[[168, 122,  78],
           │               [173, 122,  62],
           │               [170...
           └ <function RealESRGANer.enhance at 0x00000243D0A79FC0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x00000243D0A79EA0>
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2600
    │    │            └ 2048
    │    └ tensor([[[[0.3047, 0.2617, 0.2712,  ..., 0.2456, 0.2424, 0.2400],
    │                [0.2891, 0.2686, 0.2666,  ..., 0.2417, 0.2429, 0....
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

RuntimeError: The expanded size of the tensor (512) must match the existing size (552) at non-singleton dimension 2.  Target sizes: [1, 3, 512, 2048].  Tensor sizes: [3, 552, 2048]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x0000024072BF1AB0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000024072BF17E0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    │        │    └ (<weakref at 0x00000245436BA200; to 'ThreadPoolExecutor' at 0x00000243D0B86AD0>, <_queue.SimpleQueue object at 0x00000243D0BE...
    │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    └ <function _worker at 0x0000024023CF4700>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x0000024023CF4820>
    └ <concurrent.futures.thread._WorkItem object at 0x0000024577CE8070>

> File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000024577CE8070>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000024577CE8070>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000244...
             └ <concurrent.futures.thread._WorkItem object at 0x0000024577CE8070>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 427, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x0000024026A50070>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000244611843A0>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1874490>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000244611843A0>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000240791DF760>
    │    └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1874490>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000244611843A0>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1874490>
    └ <nataili.esrgan.esrgan.esrgan object at 0x00000245C2123310>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[168, 122,  78],
                   │    │       │              [173, 122,  62],
                   │    │       │              [170, 119,  61],
                   │    │       │              ...,
                   │    │       │              [162, 113,  59],
                   │    │       │              [162...
                   │    │       └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1877AC0>
                   │    └ <function esrgan.process at 0x000002407920CEE0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000245C2123310>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1877AC0>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x00000245C2123310>
                   │    └ <function esrgan.esrgan_upscale at 0x000002407920CF70>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000245C2123310>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x244C1877B80>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x00000246C52980D0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.3059, 0.2431, 0.2392,  ..., 0.2314, 0.2471, 0.2941],
             │                 [0.2667, 0.2392, 0.2275,  ..., 0.2235, 0.2275, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:29:19.028534 | worker.workers.framework:check_running_job_status:121 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:29:19.029531 | worker.workers.framework:check_running_job_status:122 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 22:30:23.081643 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (17536), thread 'ThreadPoolExecutor-0_1' (18724):
Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[  3,   9,  38],
                │    │                                [  4,   4,  43],
                │    │                                [  3,   2,  46],
                │    │                                ...,
                │    │                                [ 89, 100, 236],
                │    │                                [ 81...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0B853F0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, array([[[  3,   9,  38],
           │               [  4,   4,  43],
           │               [  3...
           └ <function RealESRGANer.enhance at 0x00000243D0A79FC0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x00000243D0A79EA0>
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2088
    │    │            └ 2048
    │    └ tensor([[[[ 0.1498,  0.1855,  0.1851,  ...,  0.9062,  0.9116,  0.9111],
    │                [ 0.1897,  0.1782,  0.1777,  ...,  0.9180, ...
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

RuntimeError: The expanded size of the tensor (0) must match the existing size (40) at non-singleton dimension 2.  Target sizes: [1, 3, 0, 2048].  Tensor sizes: [3, 40, 2048]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x0000024072BF1AB0>
    └ <Thread(ThreadPoolExecutor-0_1, started 18724)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000024072BF17E0>
    └ <Thread(ThreadPoolExecutor-0_1, started 18724)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_1, started 18724)>
    │    │        │    └ (<weakref at 0x00000245D0374E50; to 'ThreadPoolExecutor' at 0x00000243D0B86AD0>, <_queue.SimpleQueue object at 0x00000243D0BE...
    │    │        └ <Thread(ThreadPoolExecutor-0_1, started 18724)>
    │    └ <function _worker at 0x0000024023CF4700>
    └ <Thread(ThreadPoolExecutor-0_1, started 18724)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x0000024023CF4820>
    └ <concurrent.futures.thread._WorkItem object at 0x0000024461186110>

> File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000024461186110>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000024461186110>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243...
             └ <concurrent.futures.thread._WorkItem object at 0x0000024461186110>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 427, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x0000024026A50070>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243EE5BE3B0>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=512x512 at 0x24461187CD0>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243EE5BE3B0>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000240791DF760>
    │    └ <PIL.Image.Image image mode=RGB size=512x512 at 0x24461187CD0>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243EE5BE3B0>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x24461187CD0>
    └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0B853F0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[  3,   9,  38],
                   │    │       │              [  4,   4,  43],
                   │    │       │              [  3,   2,  46],
                   │    │       │              ...,
                   │    │       │              [ 89, 100, 236],
                   │    │       │              [ 81...
                   │    │       └ <PIL.Image.Image image mode=RGB size=512x512 at 0x2406F088370>
                   │    └ <function esrgan.process at 0x000002407920CEE0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0B853F0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=512x512 at 0x2406F088370>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0B853F0>
                   │    └ <function esrgan.esrgan_upscale at 0x000002407920CF70>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0B853F0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x245C2123280>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x00000246C5300670>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.1490, 0.1686, 0.1804,  ..., 0.9255, 0.8980, 0.8314],
             │                 [0.1569, 0.1882, 0.1804,  ..., 0.9137, 0.8980, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:30:25.061347 | worker.workers.framework:check_running_job_status:121 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:30:25.062346 | worker.workers.framework:check_running_job_status:122 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 22:37:36.082210 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (17536), thread 'ThreadPoolExecutor-0_1' (18724):
Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[115, 109, 105],
                │    │                                [109, 101,  98],
                │    │                                [115, 107, 101],
                │    │                                ...,
                │    │                                [130, 123, 114],
                │    │                                [123...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x00000243EE538D00>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, array([[[115, 109, 105],
           │               [109, 101,  98],
           │               [115...
           └ <function RealESRGANer.enhance at 0x00000243D0A79FC0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x00000243D0A79EA0>
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2048
    │    │            └ 0
    │    └ tensor([[[[0.3960, 0.3796, 0.3879,  ..., 0.0000, 0.0000, 0.0000],
    │                [0.4458, 0.4236, 0.4143,  ..., 0.0000, 0.0000, 0....
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

RuntimeError: The expanded size of the tensor (256) must match the existing size (296) at non-singleton dimension 3.  Target sizes: [1, 3, 2048, 256].  Tensor sizes: [3, 2048, 296]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x0000024072BF1AB0>
    └ <Thread(ThreadPoolExecutor-0_1, started 18724)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000024072BF17E0>
    └ <Thread(ThreadPoolExecutor-0_1, started 18724)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_1, started 18724)>
    │    │        │    └ (<weakref at 0x00000245D0374E50; to 'ThreadPoolExecutor' at 0x00000243D0B86AD0>, <_queue.SimpleQueue object at 0x00000243D0BE...
    │    │        └ <Thread(ThreadPoolExecutor-0_1, started 18724)>
    │    └ <function _worker at 0x0000024023CF4700>
    └ <Thread(ThreadPoolExecutor-0_1, started 18724)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x0000024023CF4820>
    └ <concurrent.futures.thread._WorkItem object at 0x00000245DC9B1270>

> File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x00000245DC9B1270>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x00000245DC9B1270>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245...
             └ <concurrent.futures.thread._WorkItem object at 0x00000245DC9B1270>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 427, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x0000024026A50070>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000024577CEA5C0>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=576x576 at 0x243EE538DC0>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000024577CEA5C0>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000240791DF760>
    │    └ <PIL.Image.Image image mode=RGB size=576x576 at 0x243EE538DC0>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x0000024577CEA5C0>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=576x576 at 0x243EE538DC0>
    └ <nataili.esrgan.esrgan.esrgan object at 0x00000243EE538D00>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[115, 109, 105],
                   │    │       │              [109, 101,  98],
                   │    │       │              [115, 107, 101],
                   │    │       │              ...,
                   │    │       │              [130, 123, 114],
                   │    │       │              [123...
                   │    │       └ <PIL.Image.Image image mode=RGB size=576x576 at 0x243D0BCDAB0>
                   │    └ <function esrgan.process at 0x000002407920CEE0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000243EE538D00>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=576x576 at 0x243D0BCDAB0>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x00000243EE538D00>
                   │    └ <function esrgan.esrgan_upscale at 0x000002407920CF70>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000243EE538D00>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x245DC9B1870>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x000002459ECA8670>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.4118, 0.3843, 0.3961,  ..., 0.4471, 0.4510, 0.4706],
             │                 [0.4000, 0.4588, 0.3961,  ..., 0.3922, 0.2980, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:37:37.933259 | worker.workers.framework:check_running_job_status:121 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:37:37.934256 | worker.workers.framework:check_running_job_status:122 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 22:37:51.669261 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (17536), thread 'ThreadPoolExecutor-0_2' (18840):
Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[237,  37,   6],
                │    │                                [255,  29,   0],
                │    │                                [255,  18,   0],
                │    │                                ...,
                │    │                                [244,  27,   0],
                │    │                                [246...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x0000024479FD9750>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, array([[[237,  37,   6],
           │               [255,  29,   0],
           │               [255...
           └ <function RealESRGANer.enhance at 0x00000243D0A79FC0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x00000243D0A79EA0>
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2048
    │    │            └ 0
    │    └ tensor([[[[-0.0117, -0.0108,  0.0212,  ...,  0.0000,  0.0000,  0.0000],
    │                [ 0.0408,  0.0115,  0.0215,  ...,  0.0000, ...
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

RuntimeError: The expanded size of the tensor (256) must match the existing size (296) at non-singleton dimension 3.  Target sizes: [1, 3, 2048, 256].  Tensor sizes: [3, 2048, 296]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x0000024072BF1AB0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000024072BF17E0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    │        │    └ (<weakref at 0x00000245436BA200; to 'ThreadPoolExecutor' at 0x00000243D0B86AD0>, <_queue.SimpleQueue object at 0x00000243D0BE...
    │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    └ <function _worker at 0x0000024023CF4700>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x0000024023CF4820>
    └ <concurrent.futures.thread._WorkItem object at 0x00000244BAF72EF0>

> File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x00000244BAF72EF0>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x00000244BAF72EF0>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243...
             └ <concurrent.futures.thread._WorkItem object at 0x00000244BAF72EF0>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 427, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x0000024026A50070>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243D0BCE650>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=576x576 at 0x24479FDB880>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243D0BCE650>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000240791DF760>
    │    └ <PIL.Image.Image image mode=RGB size=576x576 at 0x24479FDB880>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000243D0BCE650>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=576x576 at 0x24479FDB880>
    └ <nataili.esrgan.esrgan.esrgan object at 0x0000024479FD9750>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[237,  37,   6],
                   │    │       │              [255,  29,   0],
                   │    │       │              [255,  18,   0],
                   │    │       │              ...,
                   │    │       │              [244,  27,   0],
                   │    │       │              [246...
                   │    │       └ <PIL.Image.Image image mode=RGB size=576x576 at 0x24479FDB130>
                   │    └ <function esrgan.process at 0x000002407920CEE0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000024479FD9750>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=576x576 at 0x24479FDB130>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x0000024479FD9750>
                   │    └ <function esrgan.esrgan_upscale at 0x000002407920CF70>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000024479FD9750>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x24479FDB580>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x00000240792840D0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.0235, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
             │                 [0.0000, 0.0000, 0.0039,  ..., 0.0000, 0.0000, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:37:53.077494 | worker.workers.framework:check_running_job_status:121 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:37:53.079490 | worker.workers.framework:check_running_job_status:122 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 22:42:11.964494 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (17536), thread 'ThreadPoolExecutor-0_2' (18840):
Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[ 71, 209, 233],
                │    │                                [ 38, 205, 236],
                │    │                                [ 14, 190, 216],
                │    │                                ...,
                │    │                                [ 28, 143, 182],
                │    │                                [ 33...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0A6F100>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, array([[[ 71, 209, 233],
           │               [ 38, 205, 236],
           │               [ 14...
           └ <function RealESRGANer.enhance at 0x00000243D0A79FC0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x00000243D0A79EA0>
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 3112
    │    │            └ 2048
    │    └ tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
    │                [0., 0., 0.,  ..., 0., 0., 0.],
    │                [0., 0., 0.,  ..., 0., 0., 0.],...
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

RuntimeError: The expanded size of the tensor (1024) must match the existing size (1064) at non-singleton dimension 2.  Target sizes: [1, 3, 1024, 2048].  Tensor sizes: [3, 1064, 2048]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x0000024072BF1AB0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000024072BF17E0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    │        │    └ (<weakref at 0x00000245436BA200; to 'ThreadPoolExecutor' at 0x00000243D0B86AD0>, <_queue.SimpleQueue object at 0x00000243D0BE...
    │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    └ <function _worker at 0x0000024023CF4700>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x0000024023CF4820>
    └ <concurrent.futures.thread._WorkItem object at 0x0000024577CEA440>

> File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x0000024577CEA440>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x0000024577CEA440>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245...
             └ <concurrent.futures.thread._WorkItem object at 0x0000024577CEA440>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 427, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x0000024026A50070>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245A3559630>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=768x768 at 0x24603508E80>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245A3559630>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000240791DF760>
    │    └ <PIL.Image.Image image mode=RGB size=768x768 at 0x24603508E80>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245A3559630>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=768x768 at 0x24603508E80>
    └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0A6F100>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[ 71, 209, 233],
                   │    │       │              [ 38, 205, 236],
                   │    │       │              [ 14, 190, 216],
                   │    │       │              ...,
                   │    │       │              [ 28, 143, 182],
                   │    │       │              [ 33...
                   │    │       └ <PIL.Image.Image image mode=RGB size=768x768 at 0x243D0A6EEC0>
                   │    └ <function esrgan.process at 0x000002407920CEE0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0A6F100>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=768x768 at 0x243D0A6EEC0>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0A6F100>
                   │    └ <function esrgan.esrgan_upscale at 0x000002407920CF70>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x00000243D0A6F100>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x243D0BEE920>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x000002450FA3D2D0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[0.9137, 0.9255, 0.8471,  ..., 0.4941, 0.5020, 0.6039],
             │                 [0.8784, 0.9137, 0.8784,  ..., 0.4627, 0.4980, 0....
             └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:42:12.864088 | worker.workers.framework:check_running_job_status:121 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:42:12.866083 | worker.workers.framework:check_running_job_status:122 - 'RealESRGANer' object is not callable
NoneType: None
ERROR      | 2023-03-22 22:44:34.198423 | concurrent.futures.thread:run:52 - An error has been caught in function 'run', process 'MainProcess' (17536), thread 'ThreadPoolExecutor-0_2' (18840):
Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 48, in process
    output, _ = self.model["model"].enhance(img_array)
                │    │                      └ array([[[255, 255, 255],
                │    │                                [255, 255, 255],
                │    │                                [255, 255, 255],
                │    │                                ...,
                │    │                                [255, 255, 255],
                │    │                                [254...
                │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                └ <nataili.esrgan.esrgan.esrgan object at 0x0000024485CF9B40>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, array([[[255, 255, 255],
           │               [255, 255, 255],
           │               [255...
           └ <function RealESRGANer.enhance at 0x00000243D0A79FC0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 221, in enhance
    self.tile_process()
    │    └ <function RealESRGANer.tile_process at 0x00000243D0A79EA0>
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\realesrgan\utils.py", line 178, in tile_process
    self.output[:, :, output_start_y:output_end_y,
    │    │            │              └ 2600
    │    │            └ 2048
    │    └ tensor([[[[1.0361, 1.0176, 1.0098,  ..., 1.0098, 1.0107, 1.0107],
    │                [1.0127, 1.0166, 1.0107,  ..., 1.0078, 1.0078, 1....
    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

RuntimeError: The expanded size of the tensor (512) must match the existing size (552) at non-singleton dimension 2.  Target sizes: [1, 3, 512, 2048].  Tensor sizes: [3, 552, 2048]


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x0000024072BF1AB0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x0000024072BF17E0>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    │        │    └ (<weakref at 0x00000245436BA200; to 'ThreadPoolExecutor' at 0x00000243D0B86AD0>, <_queue.SimpleQueue object at 0x00000243D0BE...
    │    │        └ <Thread(ThreadPoolExecutor-0_2, started 18840)>
    │    └ <function _worker at 0x0000024023CF4700>
    └ <Thread(ThreadPoolExecutor-0_2, started 18840)>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 77, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x0000024023CF4820>
    └ <concurrent.futures.thread._WorkItem object at 0x000002454361EF50>

> File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\concurrent\futures\thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x000002454361EF50>
             │    │   │    └ ()
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x000002454361EF50>
             │    └ <bound method StableDiffusionHordeJob.start_job of <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245...
             └ <concurrent.futures.thread._WorkItem object at 0x000002454361EF50>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 427, in start_job
    self.image = post_process(post_processor, self.image, self.model_manager, strength=strength)
    │    │       │            │               │    │      │    │                       └ 0.5
    │    │       │            │               │    │      │    └ <nataili.model_manager.super.ModelManager object at 0x0000024026A50070>
    │    │       │            │               │    │      └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245A3607100>
    │    │       │            │               │    └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1875000>
    │    │       │            │               └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245A3607100>
    │    │       │            └ 'RealESRGAN_x4plus'
    │    │       └ <function post_process at 0x00000240791DF760>
    │    └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1875000>
    └ <worker.jobs.stable_diffusion.StableDiffusionHordeJob object at 0x00000245A3607100>

  File "F:\AI\dev\test\AI-Horde-Worker\worker\post_process.py", line 38, in post_process
    post_processor(input_image=image, strength=strength)
    │                          │               └ 0.5
    │                          └ <PIL.Image.Image image mode=RGB size=512x640 at 0x244C1875000>
    └ <nataili.esrgan.esrgan.esrgan object at 0x0000024485CF9B40>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\postprocessor.py", line 41, in __call__
    output_image = self.process(img, img_array, **kwargs)
                   │    │       │    │            └ {'strength': 0.5}
                   │    │       │    └ array([[[255, 255, 255],
                   │    │       │              [255, 255, 255],
                   │    │       │              [255, 255, 255],
                   │    │       │              ...,
                   │    │       │              [255, 255, 255],
                   │    │       │              [254...
                   │    │       └ <PIL.Image.Image image mode=RGB size=512x640 at 0x24485CF96F0>
                   │    └ <function esrgan.process at 0x000002407920CEE0>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000024485CF9B40>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 52, in process
    output_image = self.esrgan_upscale(model=self.model["model"], input_img=img)
                   │    │                    │    │                         └ <PIL.Image.Image image mode=RGB size=512x640 at 0x24485CF96F0>
                   │    │                    │    └ {'model': <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>, 'device': device(type='cuda', index=0), 'half_precisi...
                   │    │                    └ <nataili.esrgan.esrgan.esrgan object at 0x0000024485CF9B40>
                   │    └ <function esrgan.esrgan_upscale at 0x000002407920CF70>
                   └ <nataili.esrgan.esrgan.esrgan object at 0x0000024485CF9B40>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 150, in esrgan_upscale
    output = esrgan_enhance(model=model, img=tile)
             │                    │          └ <PIL.Image.Image image mode=RGB size=512x512 at 0x24485CF91B0>
             │                    └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>
             └ <function esrgan.esrgan_upscale.<locals>.esrgan_enhance at 0x00000245A6BA7E20>

  File "F:\AI\dev\test\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\esrgan\esrgan.py", line 63, in esrgan_enhance
    output = model(img)
             │     └ tensor([[[[1.0000, 1.0000, 1.0000,  ..., 1.0000, 1.0000, 0.9961],
             │                 [1.0000, 1.0000, 1.0000,  ..., 1.0000, 1.0000, 1....
             └ <realesrgan.utils.RealESRGANer object at 0x000002454361F5E0>

TypeError: 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:44:35.030199 | worker.workers.framework:check_running_job_status:121 - Job failed with exception, 'RealESRGANer' object is not callable
ERROR      | 2023-03-22 22:44:35.032194 | worker.workers.framework:check_running_job_status:122 - 'RealESRGANer' object is not callable
NoneType: None

@jug-dev jug-dev closed this as completed Mar 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@donaldanixon @jug-dev