RT segfaults on certain Canon CR3 files

[feuerrot]

Infos

• RT Version: current git head (2f2720b)
  • Used flags: cmake -DCMAKE_BUILD_TYPE="debug" -DBUNDLE_BASE_INSTALL_DIR="$HOME/bin/_rawtherapee" -DBUILD_BUNDLE="ON" -DPROC_TARGET_NUMBER="2" -DWITH_LTO="ON" ..
• OS: Debian testing, Kernel 5.12.0, x86_64
• Steps to reproduce:
  1. Open RT
  2. Open IMG_6588.CR3 from the filemanager
  3. RT segfaults
• I used an EF-S Lens on a Canon EOS RP, so this might be related to RT 5.7 crashes upon attempt to open Fuji GFX50R file shot in 35mm mode #5571

Output from gdb-peda:

 [----------------------------------registers-----------------------------------]
RAX: 0x3 
RBX: 0xa60 ('`\n')
RCX: 0x6 
RDX: 0xb4b4b4b4 
RSI: 0xa3b (';\n')
RDI: 0x7fff74003f10 --> 0xffffffffffffffff 
RBP: 0x7fffd0ff8730 --> 0x7fff74036650 --> 0x800000010 
RSP: 0x7fffd0ff86f0 --> 0x0 
RIP: 0x5555563ceaf6 (<DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+324>:	movzx  edx,WORD PTR [r13+0x0])
R8 : 0xca 
R9 : 0x7fffd0ff9700 (0x00007fffd0ff9700)
R10: 0x0 
R11: 0x246 
R12: 0x7fff73edc410 --> 0x0 
R13: 0x7fffda7fd000 --> 0x0 
R14: 0x7fff740365c0 --> 0x1000000010 
R15: 0x7fff740377c0 --> 0x0
EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x5555563ceaec <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+314>:	mov    esi,ecx
   0x5555563ceaee <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+316>:	mov    rdi,rax
   0x5555563ceaf1 <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+319>:	call   0x555556395b2a <_ZN5DCraw4fcolEii>
=> 0x5555563ceaf6 <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+324>:	movzx  edx,WORD PTR [r13+0x0]
   0x5555563ceafb <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+329>:	cdqe   
   0x5555563ceafd <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+331>:	mov    WORD PTR [r12+rax*2],dx
   0x5555563ceb02 <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+336>:	inc    DWORD PTR [rbp-0x28]
   0x5555563ceb05 <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+339>:	
    jmp    0x5555563cea1f <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+109>:	    jmp    0x5555563cea1f <DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)+109>
[------------------------------------stack-------------------------------------]
0000| 0x7fffd0ff86f0 --> 0x0 
0008| 0x7fffd0ff86f8 --> 0x7fffdaffc148 --> 0x7fff74003f10 --> 0xffffffffffffffff 
0016| 0x7fffd0ff8700 --> 0x7fff74003f10 --> 0xffffffffffffffff 
0024| 0x7fffd0ff8708 --> 0xa3b00000140 
0032| 0x7fffd0ff8710 --> 0xfffffffffffffbb8 
0040| 0x7fffd0ff8718 --> 0xfffffffffffffbb8 
0048| 0x7fffd0ff8720 --> 0x5555563ce9b2 (<DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void)>:	push   rbp)
0056| 0x7fffd0ff8728 --> 0x7fffdaffc148 --> 0x7fff74003f10 --> 0xffffffffffffffff 
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value
Stopped reason: SIGSEGV
0x00005555563ceaf6 in DCraw::_ZN5DCraw18crop_masked_pixelsEv._omp_fn.0(void) () at /home/feuerrot/bin/RawTherapee/rtengine/dcraw.cc:4405
4405		BAYER2(row,col) = RAW(row+top_margin,col+left_margin);

Logfiles/Image/PP3

20210524_rt.txt
IMG_6588.CR3
IMG_6588.CR3.pp3

[Thanatomanic]

Confirmed

[Thanatomanic]

You're correct that this has to do with the crop mode detection, which is not well-supported in RT. However, obviously, it has a different origin than the fixed bug in #5571

[Thanatomanic]

@feuerrot This should now be fixed. The next nightly build should be able to open your file without problems.

Side note: our camconst.json format cannot handle this kind of situation properly! One camera with multiple modes with different crops cannot be distinguished, so I had to rely on not supplying crop factors and let dcraw handle things. Less than elegant, but it works...

[feuerrot]

@Thanatomanic thank you for this really fast fix!