Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(Bug): Natron render random crashes on farm reading tiff images #862

Open
6 of 10 tasks
lpequignot opened this issue Jan 27, 2023 · 3 comments
Open
6 of 10 tasks

(Bug): Natron render random crashes on farm reading tiff images #862

lpequignot opened this issue Jan 27, 2023 · 3 comments
Labels
type:bug Something isn't working

Comments

@lpequignot
Copy link

Make sure to follow our issue report guidelines

  • I'm using the latest version of Natron (not required but recommended)
  • I've restarted Natron and the issue persists
  • I've run Natron via the command line and the issue persists
  • I've followed the contributing guidelines to the best of my understanding
  • My issue is not on the issue tracker or in a pull request already (go search for it and dig around a little bit!)
  • This bug is reproducible

Natron version

Natron 2.4.3

Operating system

Rocky linux release 8.6

System specs

RAM: 250G
CPU: AMD EPYC 75F3 32-Core Processor
GPU: Integrated Matrox G200eW3 Graphics Controller

Did you install Natron using the official installer?

  • Yes, I used the official installer
  • No, I installed from a binary archive
  • No, I compiled Natron from sources
  • No, I installed Natron via another method

Custom installation path

/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer

What were you trying to do?

  • Rendering a ntp file on farm using Deadline

Full Command: "/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer" -w Write_EXR 4701-4800 "/var/lib/Thinkbox/Deadline10/workers/rf076/jobsData/63d05790f08ce30d267e43ec/thread0_temp38RQA0/m2m_20220329_apollo_houston_io_lowpano_8972_v001.ntp"

What did you expect to happen? What happened instead?

We had random crashes, always the same traceback (provided in additional details).
We found that it was because we were using tiff file format (for STMaps).
After using exr files instead of tiff, it solved the issue.

It seems that it's a threading issue that comes form the libtiff itself used in OpenImageIO.

Maybe, an update of Oiio could fix the issue as in the latest changes there are fixes concerning libtiff threading issues.

https://github.com/OpenImageIO/oiio/blob/master/CHANGES.md

TIFF:
Guard against corrupt files with buffer overflows. (TALOS-2022-1627, CVE-2022-41977) #3628 (2.4.5/2.5.0.0)
Guard against buffer overflow for certain CMYK files. (TALOS-2022-1633, CVE-2022-41639) (TALOS-2022-1643, CVE-2022-41988) #3632 (2.4.5/2.5.0.0)
While building against the new libtiff 4.5, use its new per-tiff error handlers to ensure better thread safety. #3719 (2.5.0.0/2.4.8.0)

Step-by-step reproduction instructions

  1. rendering the ntp script on the farm

Additional details

2023-01-24 17:17:52: 0: STDOUT: Loading project: /var/lib/Thinkbox/Deadline10/workers/rf076/jobsData/63d05790f08ce30d267e43ec/thread0_temp38RQA0/m2m_20220329_apollo_houston_io_lowpano_8972_v001.ntp
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc0453c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc0453c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc119e80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc119e80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc6aca80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xc6aca80] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xca1f480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xca1f480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xcd75680] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xcd75680] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd312600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd312600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd667980] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd667980] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd9ba2c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xd9ba2c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xdf73b00] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xdf73b00] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe2d8d40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe2d8d40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe63c180] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xe63c180] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xebd8000] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xebd8000] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xef33dc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xef33dc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf27b7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf27b7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf8240c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xf8240c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfb88840] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfb88840] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfebd9c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0xfebd9c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1046bc40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1046bc40] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x107b0480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x107b0480] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x10af1600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x10af1600] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x110a6040] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x110a6040] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1140a440] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1140a440] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11760ac0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11760ac0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11d048c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x11d048c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1205c7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1205c7c0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1239ccc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x1239ccc0] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x12952200] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x12952200] UDTA parsing failed retrying raw
2023-01-24 17:17:53: 0: STDOUT: Write_EXR ==> Rendering started
2023-01-24 17:17:56: 0: STDOUT: Write_EXR ==> Frame: 4701, Progress: 1.0%, 0.3 Fps, Time Remaining: 5 minutes
2023-01-24 17:17:58: 0: STDOUT: Write_EXR ==> Frame: 4703, Progress: 2.0%, 0.4 Fps, Time Remaining: 4 minutes
2023-01-24 17:17:58: 0: STDOUT: Write_EXR ==> Frame: 4702, Progress: 3.0%, 0.5 Fps, Time Remaining: 2 minutes 56 seconds
2023-01-24 17:18:00: 0: STDOUT: Write_EXR ==> Frame: 4704, Progress: 4.0%, 0.5 Fps, Time Remaining: 2 minutes 56 seconds
2023-01-24 17:18:00: 0: STDOUT: Write_EXR ==> Frame: 4705, Progress: 5.0%, 0.7 Fps, Time Remaining: 2 minutes 21 seconds
2023-01-24 17:18:02: 0: STDOUT: Write_EXR ==> Frame: 4706, Progress: 6.0%, 0.6 Fps, Time Remaining: 2 minutes 28 seconds
2023-01-24 17:18:02: 0: STDOUT: Write_EXR ==> Frame: 4707, Progress: 7.0%, 0.7 Fps, Time Remaining: 2 minutes 5 seconds
2023-01-24 17:18:04: 0: STDOUT: Write_EXR ==> Frame: 4708, Progress: 8.0%, 0.7 Fps, Time Remaining: 2 minutes 11 seconds
2023-01-24 17:18:04: 0: STDOUT: Write_EXR ==> Frame: 4709, Progress: 9.0%, 0.8 Fps, Time Remaining: 1 minute 56 seconds
2023-01-24 17:18:06: 0: STDOUT: Write_EXR ==> Frame: 4710, Progress: 10.0%, 0.7 Fps, Time Remaining: 2 minutes 0 second
2023-01-24 17:18:06: 0: STDOUT: Write_EXR ==> Frame: 4711, Progress: 11.0%, 0.8 Fps, Time Remaining: 1 minute 48 seconds
2023-01-24 17:18:09: 0: STDOUT: Write_EXR ==> Frame: 4713, Progress: 12.0%, 0.7 Fps, Time Remaining: 1 minute 57 seconds
2023-01-24 17:18:09: 0: STDOUT: Write_EXR ==> Frame: 4712, Progress: 13.0%, 0.8 Fps, Time Remaining: 1 minute 48 seconds
2023-01-24 17:18:11: 0: STDOUT: Write_EXR ==> Frame: 4715, Progress: 14.0%, 0.8 Fps, Time Remaining: 1 minute 53 seconds
2023-01-24 17:18:11: 0: STDOUT: Write_EXR ==> Frame: 4714, Progress: 15.0%, 0.8 Fps, Time Remaining: 1 minute 45 seconds
2023-01-24 17:18:13: 0: STDOUT: Write_EXR ==> Frame: 4716, Progress: 16.0%, 0.8 Fps, Time Remaining: 1 minute 47 seconds
2023-01-24 17:18:13: 0: STDOUT: Caught segmentation fault (SIGSEGV) from thread Parallel render thread (0x7fc098004830)(0x7fc098004820), faulty address is 0x7fc1b81fec1f from 0x7fb89f227000
2023-01-24 17:18:13: 0: STDOUT: Backtrace:
2023-01-24 17:18:13: 0: STDOUT: [Frame 1]: /lib64/libc.so.6(+0xcfc1f) [0x7fc1b81fec1f]
2023-01-24 17:18:13: 0: STDOUT: [Frame 2]: /lib64/libc.so.6(+0xcfc1f) [0x7fc1b81fec1f]
2023-01-24 17:18:13: 0: STDOUT: [Frame 3]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_39TIFFInput8readspecEb+0xd62) [0x7fc1a374b002]
2023-01-24 17:18:13: 0: STDOUT: [Frame 4]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_39TIFFInput13seek_subimageEii+0x8e) [0x7fc1a374c78e]
2023-01-24 17:18:13: 0: STDOUT: [Frame 5]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_39TIFFInput4openERKSsRNS_9ImageSpecE+0x37) [0x7fc1a374cf57]
2023-01-24 17:18:13: 0: STDOUT: [Frame 6]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/../../Libraries/libOpenImageIO.so.2.3(_ZN16OpenImageIO_v2_310ImageInput4openERKSsPKNS_9ImageSpecEPNS_10Filesystem7IOProxyE+0x2cc) [0x7fc1a35747cc]
2023-01-24 17:18:13: 0: STDOUT: [Frame 7]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x15937a) [0x7fc1a5db037a]
2023-01-24 17:18:13: 0: STDOUT: [Frame 8]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x15a4ae) [0x7fc1a5db14ae]
2023-01-24 17:18:13: 0: STDOUT: [Frame 9]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x81535) [0x7fc1a5cd8535]
2023-01-24 17:18:13: 0: STDOUT: [Frame 10]: /Volumes/nas/fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/Plugins/OFX/Natron/IO.ofx.bundle/Contents/Linux-x86-64/IO.ofx(+0x1c99f3) [0x7fc1a5e209f3]
2023-01-24 17:18:13: 0: STDOUT: [Frame 11]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(ZN3OFX4Host11ImageEffect8Instance9mainEntryEPKcPKvPNS0_8Property3SetES9+0x115) [0xd44b35]
2023-01-24 17:18:13: 0: STDOUT: [Frame 12]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(ZN6Natron22OfxImageEffectInstance9mainEntryEPKcPKvPN3OFX4Host8Property3SetES9+0x48) [0x984148]
2023-01-24 17:18:13: 0: STDOUT: [Frame 13]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(_ZN3OFX4Host11ImageEffect8Instance27getRegionOfDefinitionActionEd9OfxPointDiR8OfxRectD+0x1d6) [0xd4be76]
2023-01-24 17:18:13: 0: STDOUT: [Frame 14]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(_ZN6Natron17OfxEffectInstance21getRegionOfDefinitionEmdRKNS_11RenderScaleENS_7ViewIdxEPNS_5RectDE+0x15d) [0x96d27d]
2023-01-24 17:18:13: 0: STDOUT: [Frame 15]: /fps/sw/external/redhat/natron/Natron-2.4.3-Linux-x86_64bit/bin/NatronRenderer(_ZN6Natron8ReadNode21getRegionOfDefinitionEmdRKNS_11RenderScaleENS_7ViewIdxEPNS_5RectDE+0x91) [0xa3ad11]
@lpequignot lpequignot added the type:bug Something isn't working label Jan 27, 2023
@rodlie
Copy link
Contributor

rodlie commented Jan 28, 2023

Thanks for the report.

Natron 2.5.0 on Windows and Mac includes new enough OIIO (2.4.5.0+ has tiff fixes) if i'm not mistaken, unsure about Linux. If the Linux version is off then we will update it before the next release.

@devernay
Copy link
Member

the threading-related libtiff fixes are for libtiff 4.5, and are only included in the upcoming oiio 2.5, due for summer 2023. We will upgrade both when oiio 2.5 is released

@lpequignot
Copy link
Author

Hello,

Thanks for the follow-up, we'll test the new version with oiio 2.5 when it'll be released.
In the meantime, we'll use another image format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lpequignot @devernay @rodlie