running AddressSanitizer on IlmCtlTest reports SEGV on unknown address 0x000000000008 in /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.h:9

[michaeldsmith]

add AddressSantizer build type to CMakelists.txt

set(CMAKE_CXX_FLAGS_ASAN
    "-fsanitize=address -fno-optimize-sibling-calls -fsanitize-address-use-after-scope -fno-omit-frame-pointer -g -O1"
    CACHE STRING "Flags used by the C++ compiler during AddressSanitizer builds."
    FORCE)

build unit tests with AddressSanitizer build type

cd build
cmake .. -DCMAKE_BUILD_TYPE=asan
make check

run IlmCtlTest unit test

cd ./build/unittest/IlmCtl
./IlmCtlTest 

Output is below:

-------------------------------------------------
Testing end-of-line conventions
Dos source file
Unix source file
Mac source file
ok

Testing parser
Testing variable scope
ok
Testing more variable scope
ok
Testing integer and floating-point literals
"''
 line 1
 line 2
string string one two three one two three
ok
Testing comments
ok
AddressSanitizer:DEADLYSIGNAL
=================================================================
==2075==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x7fd0caafde64 bp 0x7ffd9bd418c0 sp 0x7ffd9bd41630 T0)
==2075==The signal is caused by a READ memory access.
==2075==Hint: address points to the zero page.
    #0 0x7fd0caafde63 in Ctl::SimdInst::lineNumber() const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.h:93
    #1 0x7fd0caafde63 in Ctl::SimdInst::executePath(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:175
    #2 0x7fd0cab01603 in Ctl::SimdCallInst::execute(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:448
    #3 0x7fd0caafdd96 in Ctl::SimdInst::executePath(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:185
    #4 0x7fd0cab01603 in Ctl::SimdCallInst::execute(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:448
    #5 0x7fd0caafdd96 in Ctl::SimdInst::executePath(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:185
    #6 0x7fd0cab01603 in Ctl::SimdCallInst::execute(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:448
    #7 0x7fd0caafdd96 in Ctl::SimdInst::executePath(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:185
    #8 0x7fd0cac913ce in Ctl::SimdXContext::run(int, Ctl::SimdInst const*) /usr/src/CTL/lib/IlmCtlSimd/CtlSimdXContext.cpp:233
    #9 0x7fd0cab1332c in Ctl::SimdModule::runInitCode() /usr/src/CTL/lib/IlmCtlSimd/CtlSimdModule.cpp:122
    #10 0x7fd0ca932316 in Ctl::Interpreter::_loadModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/src/CTL/lib/IlmCtl/CtlInterpreter.cpp:383
    #11 0x7fd0ca933831 in Ctl::Interpreter::loadModuleRecursive(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/src/CTL/lib/IlmCtl/CtlInterpreter.cpp:445
    #12 0x7fd0ca933c5d in Ctl::Interpreter::loadModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/src/CTL/lib/IlmCtl/CtlInterpreter.cpp:297
    #13 0x55e12f21cd55 in testParser() /usr/src/CTL/unittest/IlmCtl/testParser.cpp:80
    #14 0x55e12f204ed8 in main /usr/src/CTL/unittest/IlmCtl/main.cpp:76
    #15 0x7fd0ca2df082 in __libc_start_main ../csu/libc-start.c:308
    #16 0x55e12f204cad in _start (/usr/src/CTL/build/unittest/IlmCtl/IlmCtlTest+0xdcad)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.h:93 in Ctl::SimdInst::lineNumber() const
==2075==ABORTING
-------------------------------------------------

[michaeldsmith]

The AddressSanitizer error does not occur if optimization is disabled by changing -O1 to -O0 in the CMakeLists asan build type as follows:

set(CMAKE_CXX_FLAGS_ASAN
    "-fsanitize=address -fno-optimize-sibling-calls -fsanitize-address-use-after-scope -fno-omit-frame-pointer -g -O0"
    CACHE STRING "Flags used by the C++ compiler during AddressSanitizer builds."
    FORCE) 

[michaeldsmith]

I've noticed some calls to alloca() in CTL, which is a function that allocates memory on the stack instead of heap. I also noticed this note about silent failure in the "bugs" section of alloca() man page https://man7.org/linux/man-pages/man3/alloca.3.html

alloca() is used by CTL in these files:

.\ctlrender\tiff_file.cc
.\ctlrender\transform.cc
.\lib\dpx\dpx_util.cc
.\lib\IlmCtl\CtlExc.cpp
.\lib\IlmCtl\CtlTypeStorage.cpp

As described above, AddressSantizer gives SEGV error (which is the same error mentioned in the bugs section of alloca() man page) when running on the CTL's unit test IlmCtlTest,

=================================================================
==2075==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x7fd0caafde64 bp 0x7ffd9bd418c0 sp 0x7ffd9bd41630 T0)
==2075==The signal is caused by a READ memory access.
==2075==Hint: address points to the zero page.
    #0 0x7fd0caafde63 in Ctl::SimdInst::lineNumber() const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.h:93
    #1 0x7fd0caafde63 in Ctl::SimdInst::executePath(Ctl::SimdBoolMask&, Ctl::SimdXContext&) const /usr/src/CTL/lib/IlmCtlSimd/CtlSimdInst.cpp:175
...

[michaeldsmith]

The reported AddressSanitizer error does not occur if lines 61-62 of file /lib/IlmCtlSimd/CtlSimdInst.cpp are changed

from:

#include <CtlSimdInst.h>
#include <sstream>

to:

#pragma GCC push_options
#pragma GCC optimize ("O0")
#include <CtlSimdInst.h>
#pragma GCC pop_options
#include <sstream>

to disable the compiler's optimization of CtlSimdInst.h

[michaeldsmith]

The reported AddressSanitizer error does not occur if lines 87-88 of file /lib/IlmCtlSimd/CtlSimdInst.h are compiled without optimization by changing

    void		executePath (SimdBoolMask &mask,
				     SimdXContext &xcontext) const;

to

#pragma GCC push_options
#pragma GCC optimize ("O0")
    void		executePath (SimdBoolMask &mask,
				     SimdXContext &xcontext) const;
#pragma GCC pop_options

[michaeldsmith]

The reported AddressSanitizer error occurs when unittest IlmCtlTest processes line 96 of /unittest/IlmCtl/testVSArrays.ctl that calls the CTL code function empty()

int emptyA[1][2][3];
empty(emptyA);

CTL function empty() on lines 56-60 /unittest/IlmCtl/testVSArrays.ctl is an empty ctl function

void
empty(int a[][][])
{

}

So perhaps the compiler optimization when using -O1 or higher is causing the useless function empty() to be eliminated, thus the function pointer is zero, and disabling the compiler optimization with -O0 keeps the empty function and thus the function pointer is not zero.

[michaeldsmith]

To avoid attempting to call a function that has been optimized out, like CTL function empty() on lines 56-60 /unittest/IlmCtl/testVSArrays.ctl , a check can be added to SimdCallInst::execute() before calling executePath() at line 449 of file /lib/IlmCtlSimd/CtlSimdInst.cpp

for example the line:

_callPath->executePath (callMask, xcontext);

can be changed to:

if( 0 != _callPath ) 
{
	_callPath->executePath (callMask, xcontext);
}

[michaeldsmith]

A similar type of check can be added to SimdCallBranchInst::execute() on line 240 of /lib/IlmCtlSimd/CtlSimdInst.cpp

where _truePath->execute(mask, xcontext) appears, change it to

if( 0 != _truePath )
     _truePath->executePath (mask, xcontext);

where _falsePath->execute(mask, xcontext) appears, change it to

if( 0 != _falsePath )
     _falsePath->executePath (mask, xcontext);