Writing to cout in external functions called to compute package constants crashes OMEdit

[casella]

We are experiencing a weird but fatal bug with ExternalMedia in OpenModelica. The bug is present both in the Linux and Windows versions, but is more easily reproduced under Windows.

Steps to Reproduce

• Download the latest development version of ExternalMedia here.
• Move the libExternalMediaLib.so and libExternalMediaLib.so.4.0.0 files one directory up from Resources/Library/linux64/gcc93, so that OpenModelica finds them (this is a separate issue of the library CMAKE file, never mind that here).
• Open the ExternalMedia package and open the ExternalMedia.Test.CoolProp.CO2.TestStatesSatSubcritical model
• Press the CheckModel button

On Linux, this causes an immediate crash. On Windows, the program doesn't crash immediately, but is very slow and eventually crashes afterwards at some random point in time, due to corrupted memory.

Analysis of the stack trace reveals the following steps:

• the checkModel API function is called
• the frontend evaluates the package constants, which are bound to external functions, which are available in the shared library libExternalMediaLib.so
• the external functions eventually call the CoolPropSolver C++ object constructor
• this constructor tries to write stuff to cout, causing OMEdit to crash

@fedetft has played around a bit with that chunk of code. It turns out that if you try to print out stuff using printf or ofstream, that works fine, but as soon as you try to output stuff with cout, OMEdit crashes. It also turns out that if you do checkModel from a script, it runs fine

So, the problem seems to be that if the frontend calls a dynamically linked external function for constant evaluation, that in turn calls a C++ function using cout, that thing is not handled well by OMEdit.

I'm not sure how exactly the console output of omc is redirected to OMEdit's various log windows, but apparently something there doesn't work well with cout in this case.

Note that @fedetft has written other shared libraries using cout, that run without any problem as long as they are called at runtime. Apparently, this issue only pops up when they are called by the frontend at compile time.

@perost, @adrpo, @sjoelund, @adeas31, @mahge, do you have any clue about what could be the reason of this bug? This is a serious problem, because it actually prevents us from using ExternalMedia in OMEdit, which is a big deal, as many users (including ourselves) badly need this library to work in OpenModelica.

[adeas31]

I guess the binaries copy step is not required for Windows, right?

For me it crashes inside the GC in function GC_acquire_mark_lock. Here is the complete backtrace.txt

[perost]

I guess the binaries copy step is not required for Windows, right?

Doesn't seem necessary on Linux either, I guess we've fixed that.

For me it crashes inside the GC in function GC_acquire_mark_lock. Here is the complete backtrace.txt

Looks more like SystemImpl__freeLibrary if you look at the trace for thread 1, or am I missing something?

On Linux I get:

Click to expand

0x00007fffed78e48a in std::ostream::sentry::sentry (this=0x7fffffff9060, __os=...) at /build/gcc/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:51
  #0  0x00007fffed78e48a in std::ostream::sentry::sentry(std::ostream&) (this=0x7fffffff9060, __os=...)
    at /build/gcc/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:51
#1  0x00007fffed78ec9d in std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) (__out=..., __s=0x55555ce91b00 "calc_transport", __n=14)
    at /build/gcc/src/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:83
#2  0x00007ffee2c7a3d0 in CoolPropSolver::CoolPropSolver(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
    at /home/per/workspace/OpenModelica/Models/ExternalMedia 4.0.0/Resources/Library/linux64/libExternalMediaLib.so
#3  0x00007ffee2c8fb92 in SolverMap::getSolver(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
    at /home/per/workspace/OpenModelica/Models/ExternalMedia 4.0.0/Resources/Library/linux64/libExternalMediaLib.so
#4  0x00007ffee2c86379 in TwoPhaseMedium_getMolarMass_C_impl ()
    at /home/per/workspace/OpenModelica/Models/ExternalMedia 4.0.0/Resources/Library/linux64/libExternalMediaLib.so
#5  0x00007ffff192feaa in ffi_call_unix64 ()
    at /home/per/workspace/OpenModelica/OMCompiler/3rdParty/libffi/src/x86/unix64.S:105
#6  0x00007ffff192f7db in ffi_call_int
    (cif=0x7fffffff9830, fn=0x7ffee2c86210 <TwoPhaseMedium_getMolarMass_C_impl>, rvalue=0x7fff05a4f670, avalue=0x7ffeea3d6240, closure=0x0) at /home/per/workspace/OpenModelica/OMCompiler/3rdParty/libffi/src/x86/ffi64.c:669
#7  0x00007ffff192f858 in ffi_call
    (cif=0x7fffffff9830, fn=0x7ffee2c86210 <TwoPhaseMedium_getMolarMass_C_impl>, rvalue=0x7fff05a4f670, avalue=0x7ffeea3d6240) at /home/per/workspace/OpenModelica/OMCompiler/3rdParty/libffi/src/x86/ffi64.c:688
#8  0x00007ffff191cb4e in FFI_callFunction
    (fnHandle=1, args=0x7fff0d58df63, specs=0x7fff0d58df33, returnType=0x7ffff2c64c73 <_OMC_LIT_STRUCT48+3>, outputArgs=0x7fffffff98c0) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/runtime/ffi_omc.c:596
#9  0x00007ffff1812c52 in omc_FFI_callFunction
    (threadData=0x7fffffffe200, _fnHandle=1, _args=0x7fff0d58df63, _specs=0x7fff0d58df33, _returnType=0x7ffff2c64c73 <_OMC_LIT_STRUCT48+3>, out_outputArgs=0x7fffffff9960) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Util/FFI.mo:11
#10 0x00007ffff0dc9978 in omc_NFEvalFunction_callExternalFunction
    (threadData=0x7fffffffe200, _extName=0x7fff2ef7df83, _fn=0x7fff0d57c103, _args=0x7ffff1b5a563 <mmc_nil+3>, _extArgs=0x7ffeec9459c3, _outputRef=0x7fff0d583703, _extAnnotation=0x7ffeebaeac23, _debug=0 '\000')
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFEvalFunction.mo:1148
#11 0x00007ffff0dcff1a in omc_NFEvalFunction_evaluateExternal
    (threadData=0x7fffffffe200, _fn=0x7fff0d57c103, _args=0x7ffff1b5a563 <mmc_nil+3>, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFEvalFunction.mo:165
#12 0x00007ffff0dd05ab in omc_NFEvalFunction_evaluate
    (threadData=0x7fffffffe200, _fn=0x7fff0d57c103, _args=0x7ffff1b5a563 <mmc_nil+3>, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFEvalFunction.mo:83
#13 0x00007ffff0d5fd5b in omc_NFCeval_evalNormalCall
    (threadData=0x7fffffffe200, _fn=0x7fff0d57c103, _args=0x7ffff1b5a563 <mmc_nil+3>, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>) at /home/per/workspace/OpenModelica/build/OMCompiler/Compiler/c_files/NFCeval.c:6949
#14 0x00007ffff0d5fe6e in omc_NFCeval_evalNormalCallExp
    (threadData=0x7fffffffe200, _callExp=0x7ffeec96a883, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:1951
#15 0x00007ffff0d61e8d in closure5_NFCeval_evalNormalCallExp
   c96a883)
    at /home/per/workspace/OpenModelica/build/OMCompiler/Compiler/c_files/NFCeval.c:7545
#16 0x00007ffff0dde7eb in omc_NFExpression_mapSplitExpressions (threadData=0x7fffffffe200, _exp=0x7ffeec96a883, _func=0x7ffeec96a803)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFExpression.mo:5561
#17 0x00007ffff0d621be in omc_NFCeval_evalCall (threadData=0x7fffffffe200, _call=0x7fff0d58b913, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:1825
#18 0x00007ffff0d6db8c in omc_NFCeval_evalExp (threadData=0x7fffffffe200, __omcQ_24in_5Fexp=0x7ffeec964d03, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:212
#19 0x00007ffff0d6ad19 in omc_NFCeval_makeRecordFieldBindingFromParent
    (threadData=0x7fffffffe200, _cref=0x7fff0d583c03, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:739
#20 0x00007ffff0d6afd0 in omc_NFCeval_makeComponentBinding
    (threadData=0x7fffffffe200, _component=0x7fff0d5760c3, _node=0x7fff024b7003, _cref=0x7fff0d583c03, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:677
#21 0x00007ffff0d6c813 in omc_NFCeval_evalComponentBinding
    (threadData=0x7fffffffe200, _node=0x7fff024b7003, _cref=0x7fff0d583c03, _defaultExp=0x7fff0d585e43, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>, _evalSubscripts=1 '\001', _liftExp=1 '\001') at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:399
#22 0x00007ffff0d6d168 in omc_NFCeval_evalCref
    (threadData=0x7fffffffe200, _cref=0x7fff0d583c03, _defaultExp=0x7fff0d585e43, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>, _evalSubscripts=1 '\001', _liftExp=1 '\001') at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:364
#23 0x00007ffff0d6d7cf in omc_NFCeval_evalExp (threadData=0x7fffffffe200, __omcQ_24in_5Fexp=0x7fff0d585e43, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:185
#24 0x00007ffff0d6ad19 in omc_NFCeval_makeRecordFieldBindingFromParent
    (threadData=0x7fffffffe200, _cref=0x7fff0d583e43, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:739
#25 0x00007ffff0d6afd0 in omc_NFCeval_makeComponentBinding
    (threadData=0x7fffffffe200, _component=0x7fff07d8f543, _node=0x7fff02430e43, _cref=0x7fff0d583e43, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:677
#26 0x00007ffff0d6c813 in omc_NFCeval_evalComponentBinding
    (threadData=0x7fffffffe200, _node=0x7fff02430e43, _cref=0x7fff0d583e43, _defaultExp=0x7fff0d582d53, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>, _evalSubscripts=0 '\000', _liftExp=1 '\001') at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:399
#27 0x00007ffff0d6d168 in omc_NFCeval_evalCref
    (threadData=0x7fffffffe200, _cref=0x7fff0d583e43, _defaultExp=0x7fff0d582d53, _target=0x7ffff2b4b463 <_OMC_LIT_STRUCT1+3>, _evalSubscripts=0 '\000', _liftExp=1 '\001') at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFCeval.mo:364
#28 0x00007ffff0dbb3af in omc_NFEvalConstants_evaluateFuncExpTraverser
    (threadData=0x7fffffffe200, _exp=0x7fff0d582db3, _fnNode=0x7fff043f4193, _evaluateAll=0 '\000', _changed=0 '\000', out_outChanged=0x0)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFEvalConstants.mo:603
#29 0x00007ffff0dbb6c3 in omc_NFEvalConstants_evaluateFuncExp (threadData=0x7fffffffe200, _exp=0x7fff0d582db3, _fnNode=0x7fff043f4193, _evaluateAll=0 '\000')
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFEvalConstants.mo:581
#30 0x00007ffff0dbb713 in boxptr_NFEvalConstants_evaluateFuncExp (threadData=0x7fffffffe200, _exp=0x7fff0d582db3, _fnNode=0x7fff043f4193, _evaluateAll=0x0)
    at /home/per/workspace/OpenModelica/build/OMCompiler/Compiler/c_files/NFEvalConstants.c:322
#31 0x00007ffff0dbb761 in closure1_NFEvalConstants_evaluateFuncExp (thData=0x7fffffffe200, closure=0x7ffeec92b803, exp=0x7fff0d582db3)
    at /home/per/workspace/OpenModelica/build/OMCompiler/Compiler/c_files/NFEvalConstants.c:331
#32 0x00007ffff0eb6a9a in omc_NFStatement_mapExp (threadData=0x7fffffffe200, __omcQ_24in_5Fstmt=0x7fff0d583fc3, _func=0x7ffeec92b703)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFStatement.mo:440
#33 0x00007ffff0eb7696 in omc_NFStatement_mapExpList (threadData=0x7fffffffe200, __omcQ_24in_5Fstmtl=0x7ffeec92b963, _func=0x7ffeec92b703)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFStatement.mo:421
#34 0x00007ffff0d15628 in omc_NFAlgorithm_mapExp (threadData=0x7fffffffe200, __omcQ_24in_5Falg=0x7fff0d581183, _func=0x7ffeec92b703)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFAlgorithm.mo:118
#35 0x00007ffff0d15508 in omc_NFAlgorithm_mapExpList (threadData=0x7fffffffe200, __omcQ_24in_5Falgs=0x7ffeec92b923, _func=0x7ffeec92b703)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFAlgorithm.mo:129
#36 0x00007ffff0ea03db in omc_NFSections_mapExp (threadData=0x7fffffffe200, __omcQ_24in_5Fsections=0x7fff0d583f83, _mapFn=0x7ffeec92b703)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFSections.mo:288
#37 0x00007ffff0e28a04 in omc_NFFunction_Function_mapExp
    (threadData=0x7fffffffe200, __omcQ_24in_5Ffn=0x7fff0d57c303, _mapFn=0x7ffeec92b703, _mapFnFields=0x7ffeec92b743, _mapParameters=1 '\001', _mapBody=1 '\001') at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFunction.mo:1891
#38 0x00007ffff0dbb918 in omc_NFEvalConstants_evaluateFunction (threadData=0x7fffffffe200, __omcQ_24in_5Ffunc=0x7fff0d57c303)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFEvalConstants.mo:559
#39 0x00007ffff0e0f91e in omc_NFFlatten_flattenFunction (threadData=0x7fffffffe200, _func=0x7fff07d83903, __omcQ_24in_5Ffuncs=0x7fff0d581403)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFlatten.mo:2173
#40 0x00007ffff0e0fbc6 in omc_NFFlatten_collectExpFuncs__traverse (threadData=0x7fffffffe200, _exp=0x7ffeecd9c003, __omcQ_24in_5Ffuncs=0x7fff0d581403)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFlatten.mo:2137
#41 0x00007ffff0df0094 in omc_NFExpression_fold
    (threadData=0x7fffffffe200, _exp=0x7ffeecd9c003, _func=0x7ffff2bb8983 <boxvar_lit_NFFlatten_collectExpFuncs__traverse+3>, _arg=0x7fff0d581403)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFExpression.mo:3143
--Type <RET> for more, q to quit, c to continue without paging--
#42 0x00007ffff0e0fceb in omc_NFFlatten_collectExpFuncs (threadData=0x7fffffffe200, _exp=0x7ffeecd9c003, __omcQ_24in_5Ffuncs=0x7fff0d581403)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFlatten.mo:2124
#43 0x00007ffff0e108a2 in omc_NFFlatten_collectBindingFuncs (threadData=0x7fffffffe200, _binding=0x7fff05c41303, __omcQ_24in_5Ffuncs=0x7fff0d581403)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFlatten.mo:1887
#44 0x00007ffff0e10974 in omc_NFFlatten_collectComponentFuncs (threadData=0x7fffffffe200, _var=0x7fff05c4db63, __omcQ_24in_5Ffuncs=0x7fff0d581403)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFlatten.mo:1867
#45 0x00007ffff0770dd7 in omc_List_fold
    (threadData=0x7fffffffe200, _inList=0x7ffeecd28b03, _inFoldFunc=0x7ffff2bb8ac3 <boxvar_lit_NFFlatten_collectComponentFuncs+3>, _inStartValue=0x7ffff2bb9c83 <_OMC_LIT_STRUCT98+3>) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Util/List.mo:3584
#46 0x00007ffff0e1b4d5 in omc_NFFlatten_collectFunctions (threadData=0x7fffffffe200, _flatModel=0x7fff0d5771e3)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFFlatten.mo:202
#47 0x00007ffff0e50c2d in omc_NFInst_instClassInProgram
    (threadData=0x7fffffffe200, _classPath=0x7ffeebaaf5d3, _program=0x7fff0202b803, _dumpFlat=0 '\000', out_functions=0x7fffffffb430, out_flatString=0x7fffffffb438) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/NFFrontEnd/NFInst.mo:191
#48 0x00007ffff1000765 in omc_CevalScriptBackend_runFrontEndWorkNF
    (threadData=0x7fffffffe200, _className=0x7ffeebaaf5d3, _dumpFlat=0 '\000', out_functions=0x7fffffffb5d8, out_flatString=0x7fffffffb5b8)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/CevalScriptBackend.mo:3387
#49 0x00007ffff1000b85 in omc_CevalScriptBackend_runFrontEndWork
    (threadData=0x7fffffffe200, _inCache=0x7ffeebaadb43, _inEnv=0x7ffff22e2923 <_OMC_LIT_STRUCT68+3>, _className=0x7ffeebaaf5d3, _relaxedFrontEnd=0 '\000', _dumpFlat=0 '\000', out_env=0x7fffffffb7a8, out_dae=0x7fffffffb7b8, out_flatString=0x7fffffffb7b0)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/CevalScriptBackend.mo:3302
#50 0x00007ffff10018d5 in omc_CevalScriptBackend_runFrontEnd
    (threadData=0x7fffffffe200, __omcQ_24in_5Fcache=0x7ffeebaadb43, __omcQ_24in_5Fenv=0x7ffff22e2923 <_OMC_LIT_STRUCT68+3>, _className=0x7ffeebaaf5d3, _relaxedFrontEnd=0 '\000', _dumpFlat=0 '\000', _transform=0 '\000', out_env=0x7fffffffb950, out_odae=0x7fffffffb948, out_flatString=0x0)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/CevalScriptBackend.mo:3226
#51 0x00007ffff0fec662 in omc_CevalScriptBackend_checkModel
    (threadData=0x7fffffffe200, __omcQ_24in_5Fcache=0x7ffeebaadb43, _inEnv=0x7ffff22e2923 <_OMC_LIT_STRUCT68+3>, _className=0x7ffeebaaf5d3, _inMsg=0x7ffff2dd96b3 <_OMC_LIT_STRUCT4+3>, out_outValue=0x7fffffffbc18) at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/CevalScriptBackend.mo:5885
#52 0x00007ffff1027ba1 in omc_CevalScriptBackend_cevalInteractiveFunctions3
    (threadData=0x7fffffffe200, _inCache=0x7ffeebaadb43, _inEnv=0x7ffff22e2923 <_OMC_LIT_STRUCT68+3>, _inFunctionName=0x7ffff1b874a3 <_OMC_LIT_STRUCT95+3>, _inVals=0x7ffeebab4863, _msg=0x7ffff2dd96b3 <_OMC_LIT_STRUCT4+3>, out_outValue=0x7fffffffc7d0)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/CevalScriptBackend.mo:1296
#53 0x00007ffff0414e49 in omc_CevalScript_cevalInteractiveFunctions2
    (threadData=0x7fffffffe200, _cache=0x7ffeebaadb43, _env=0x7ffff22e2923 <_OMC_LIT_STRUCT68+3>, _functionName=0x7ffff1b874a3 <_OMC_LIT_STRUCT95+3>, _args=0x7ffeebab4863, _msg=0x7ffff2dd96b3 <_OMC_LIT_STRUCT4+3>, out_outValue=0x7fffffffd000)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/CevalScript.mo:1476
#54 0x00007ffff0faed6c in omc_OpenModelicaScriptingAPI_checkModel (threadData=0x7fffffffe200, _className=0x7ffeeb81b193)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/OpenModelicaScriptingAPI.mo:2095
#55 0x0000555555afe0ca in OMCInterface::checkModel(QString) (this=0x5555569c8f60, className=...)
    at /home/per/workspace/OpenModelica/OMCompiler/Compiler/Script/OpenModelicaScriptingAPIQt.cpp:8220
#56 0x000055555577e0bb in OMCProxy::checkModel(QString) (this=0x5555569da910, className=...)
    at /home/per/workspace/OpenModelica/OMEdit/OMEditLIB/OMC/OMCProxy.cpp:2416
#57 0x000055555573dc6f in MainWindow::checkModel(LibraryTreeItem*) (this=0x5555566dd6b0, pLibraryTreeItem=0x55555b729cf0)
    at /home/per/workspace/OpenModelica/OMEdit/OMEditLIB/MainWindow.cpp:863
#58 0x00005555557491f4 in MainWindow::checkModel() (this=0x5555566dd6b0) at /home/per/workspace/OpenModelica/OMEdit/OMEditLIB/MainWindow.cpp:2270
#59 0x0000555555b34aa3 in MainWindow::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)
    (_o=0x5555566dd6b0, _c=QMetaObject::InvokeMetaMethod, _id=39, _a=0x7fffffffd4e0)
    at /home/per/workspace/OpenModelica/build/OMEdit/OMEditLIB/OMEditLib_autogen/EWIEGA46WW/moc_MainWindow.cpp:489
#60 0x00007ffff378a800 in  () at /usr/lib/libQt5Core.so.5
#61 0x00007ffff4765203 in QAction::triggered(bool) () at /usr/lib/libQt5Widgets.so.5
#62 0x00007ffff4767f18 in QAction::activate(QAction::ActionEvent) () at /usr/lib/libQt5Widgets.so.5
#63 0x00007ffff4865542 in  () at /usr/lib/libQt5Widgets.so.5
#64 0x00007ffff4865695 in QAbstractButton::mouseReleaseEvent(QMouseEvent*) () at /usr/lib/libQt5Widgets.so.5
#65 0x00007ffff496361b in QToolButton::mouseReleaseEvent(QMouseEvent*) () at /usr/lib/libQt5Widgets.so.5
#66 0x00007ffff47af0be in QWidget::event(QEvent*) () at /usr/lib/libQt5Widgets.so.5
#67 0x00007ffff476bd62 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () at /usr/lib/libQt5Widgets.so.5
#68 0x00007ffff4773ac9 in QApplication::notify(QObject*, QEvent*) () at /usr/lib/libQt5Widgets.so.5
#69 0x00007ffff375341a in QCoreApplication::notifyInternal2(QObject*, QEvent*) () at /usr/lib/libQt5Core.so.5
#70 0x00007ffff477257b in QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool) ()
    at /usr/lib/libQt5Widgets.so.5
#71 0x00007ffff47c8a84 in  () at /usr/lib/libQt5Widgets.so.5
#72 0x00007ffff47cbdb5 in  () at /usr/lib/libQt5Widgets.so.5
#73 0x00007ffff476bd62 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () at /usr/lib/libQt5Widgets.so.5
#74 0x00007ffff375341a in QCoreApplication::notifyInternal2(QObject*, QEvent*) () at /usr/lib/libQt5Core.so.5
#75 0x00007ffff406f1f0 in QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) () at /usr/lib/libQt5Gui.so.5
--Type <RET> for more, q to quit, c to continue without paging--
#76 0x00007ffff40447d5 in QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) () at /usr/lib/libQt5Gui.so.5
#77 0x00007fffe02bdfcc in  () at /usr/lib/libQt5XcbQpa.so.5
#78 0x00007fffebab74dc in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#79 0x00007fffebb0b799 in  () at /usr/lib/libglib-2.0.so.0
#80 0x00007fffebab4bc1 in g_main_context_iteration () at /usr/lib/libglib-2.0.so.0
#81 0x00007ffff37ac046 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () at /usr/lib/libQt5Core.so.5
#82 0x00007ffff3751d8c in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () at /usr/lib/libQt5Core.so.5
#83 0x00007ffff375a2f4 in QCoreApplication::exec() () at /usr/lib/libQt5Core.so.5
#84 0x000055555570bff0 in main(int, char**) (argc=1, argv=0x7fffffffe448) at /home/per/workspace/OpenModelica/OMEdit/OMEditGUI/main.cpp:178

I have no clue why that happens though, something to do with how OMEdit captures stdout maybe? I wonder if the same thing happens if printf is used instead of std::cout.

[perost]

I wonder if the same thing happens if printf is used instead of std::cout.

It does not, using printf seems to work just fine and the output ends up in omeditoutput.txt. So the issue is specifically with std::cout and not with stdout, but I don't really have any clue what it might be.

[fedetft]

Here's a stacktrace where it crashes instantly, instead of corrupting memory and crashing after a while. I was debugging yesterday with @casella and it was repeatable most of the time on Linux with 1.19.0~dev-697-gfa7189b

openmodelica.stacktrace.OMEdit.txt

As I was set up for recompiling the shared object of ExternalMedia I coud run some tests:

• doing a printf does not crash, but I could not see the output of printf, neither on the shell I was running OMEdit from nor from the OMEdit GUI
• opening files and writing into them, either with C FILE* functione or with C++ ofstream classes it does not crash
• any cout made it crash.

We tried to add code to ExternalMedia that created a first file, then did a cout<<"Hello"<<endl; and then created a second file. While checking the model that caused the call of that C++ function, the first file was created, then OMEdit crashed and the second file was not created, so it crashes right at the cout.

[adeas31]

I have no clue why that happens though, something to do with how OMEdit captures stdout maybe?

https://github.com/OpenModelica/OpenModelica/blob/master/OMEdit/OMEditLIB/MainWindow.cpp#L141

[mahge]

I guess the binaries copy step is not required for Windows, right?
Doesn't seem necessary on Linux either, I guess we've fixed that.

If you do not copy the binaries then we will not find them. It should print warnings saying that the library is not found in any of the searched directories. However, checkModel seems to pass and the model reaches all the way to codegen (if you try to simulate) without having to evaluate the external function. So maybe the evaluation is not strictly necessary? is that possible @perost? Also I would also expect it not to crash for you if the libraries are not copied. There is something more here.

If I copy the library as @casella said then I get the same trace as @perost.

It does not, using printf seems to work just fine and the output ends up in omeditoutput.txt. So the issue is specifically with std::cout and not with stdout, but I don't really have any clue what it might be.

did you manage to rebuild the library and verify it? I am looking at the repository and I can not see where the shared libraries are built. I also have no clue as what it might be but I want to see what std::cout in the library and std::cout in OMEdit point to. It might help if we can see how the libraries are built.

[perost]

The reason why the NF calls the external functions is because e.g. ExternalTwoPhaseMedium.molarMass uses fluidConstants[1].molarMass, and if it doesn't evaluate that we can't generate valid code for it since the code generator doesn't handle package constants in functions well. As pointed out by @mahge the NF itself doesn't actually need to evaluate those functions, but then we can't actually build and simulate the model.

And even if we could avoid evaluate them in this case the general issue would still remain, since there are situations where the NF really must evaluate external functions (e.g. if they're used to determine dimension sizes of variables).

[fedetft]

If you are on Linux building ExternalMedia is relatively easy.
Clone the externalMedia repository and in the top level directory do a

cmake -B build -S Projects -DCMAKE_BUILD_TYPE=Release
cmake -B build -S Projects -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target install

(ExternalMedia documentation says to run the first command twice, no idea why though)

[mahge]

If you are on Linux building ExternalMedia is relatively easy.
Clone the externalMedia repository and in the top level directory do a

This will only build a static library. I am guessing this is not how the libraries in the distribution were generated. Maybe I am missing something else.

[perost]

If you are on Linux building ExternalMedia is relatively easy.
Clone the externalMedia repository and in the top level directory do a

This will only build a static library. I am guessing this is not how the libraries in the distribution were generated. Maybe I am missing something else.

Check out the v4-dev branch instead of master.

[perost]

I have no clue why that happens though, something to do with how OMEdit captures stdout maybe?

https://github.com/OpenModelica/OpenModelica/blob/master/OMEdit/OMEditLIB/MainWindow.cpp#L141

Removing that made no difference either, so OMEdit capturing stdout does not seem to be the issue.

[fedetft]

Yes I forgot that you also have to checkout v4-dev, thanks @perost

[fedetft]

By the way, I tried adding a cout in a C++ function of another Modelica library we have in development, call that function in a package constant, and OMedit crashes with a similar stacktrace. So the bug is not specific to ExternalMedia

[casella]

The reason why the NF calls the external functions is because e.g. ExternalTwoPhaseMedium.molarMass uses fluidConstants[1].molarMass, and if it doesn't evaluate that we can't generate valid code for it since the code generator doesn't handle package constants in functions well. As pointed out by @mahge the NF itself doesn't actually need to evaluate those functions, but then we can't actually build and simulate the model.

And even if we could avoid evaluate them in this case the general issue would still remain, since there are situations where the NF really must evaluate external functions (e.g. if they're used to determine dimension sizes of variables).

BTW, evaluating this constant takes quite some time, so it is also essential for efficiency that this is done at compile time.

[fedetft]

I've prepared a minimum non working example that will hopefully simplify the debugging. I only tested it on Linux, but the Resources/Source directory contains a CMakeLists.txt that should also work on Windows.
Just checking the TestFoo model makes OMEdit crash, and recompiling the library without the cout makes it work.
mnwe.tar.gz

[albertoleva]

The example by Federico (fedetft) runs on my stable 1.18.1.
Whatever the problem is, seems it has been introduced in some recent nightly.
HTH

[fedetft]

I checked as well, and I confirm that with the release 1.18.1 the minimum non-working example runs, so it appears the cout bug is a regression introduced in 1.19

[casella]

@perost and @mahger analyzed the issue, but it's still not clear what is the root cause.

The issue is only present in OMEdit, not with OMShell, so it's some issue with OMEdit or QT. Maybe QT is redirecting some stuff and messing with them. Could be OMSimulator, which is also linked to OMEdit. The bug is present both with gcc and clang, so it's not a compiler issue. In any case, it seems that cout for some reason is not initialized or redirected properly.

@mahge will try compiling without OMSimulator and see what happens.

[fedetft]

The cout object may be corrupted in some way, if someone can attach to OMEdit with GDB, it may be possible to dump the cout object bytes right at the beginning of main() and after OMEdit has started to see if there is any obvious corruption, such as the memory area being zeroed out.

[casella]

@mahge, did it work without OMSimulator?

I will do some bisection with the nightly builds and try to figure out when things got wrong

[perost]

I will do some bisection with the nightly builds and try to figure out when things got wrong

I assume that would be when we implemented evaluation of external functions via libffi in the first place. Before that the MWE would just fail with an error message, since it uses an external function to determine the size of an array.

Also, I did run OMEdit through valgrind earlier in the week, but it didn't show any illegal memory accesses (besides the usual GC noise) before it tries to use cout and crashes.

[fedetft]

Don't know the exact details, but the fact that the address of cout in the build causing problems is 0xcfee80 (which looks like a relative address needing relocation) and the one working is 0x7ffff0d3d480 which seems a correctly relocated address, I'm not surprised the dynamic linker does something along the line of "Ah, I see you linked libstdc++ statically, so I don't need to do relocations at runtime at all", and does this also when dlopen loads a .so that depends on libstdc++, leaving the GOT entry for cout unrelocated...

Does -fPIC force data accesses to go through the GOT? I don't remember, this may be why the dynamic linker springs back into action and starts doing its job. In any case, my advice remains the same: don't link listdc++ statically if your code can load shared objects.

[casella]

Good that you found out what the problem is. Never underestimate the power of bisection. Sectumsempra! 😄

[casella]

Discussion at devmeeting. Remove flags to link libstdc++ from all over the place (including OMEdit!), so it's linked dynamically. The library is already copied in the proper place, so there shouldn't be any further problems. Should work on Windows and Linux.

@mahge will try this out and test.

Also make sure that ExternalMedia links dynamically

[mahge]

Alright so after a long campaign of compiling and recompiling and pulling hair we have almost found what causes these segfaults and how we can possibly avoid them.

Spoiler alert: it is two different segfaults. Two different segfaults for two different reasons on Windows vs Linux.

The linux one is due to std::cout as we have already found out with possible explanation given by @fedetft above. The solution to this is to either use the shared version of libantlr4 or compile the static library with position independent code. My suggestion is to compile the static library as position independent. In fact we should compile all static libraries we have that way. The CMake build already compiles all static libraries as PIC, it just did not do it for the GUI related libraries.

The Windows segfault has nothing to do with std::cout. It has nothing to do with OMEdit (it will happen on the CLI). It also has nothing to do specifically with ExternalMedia for that matter.

OMC was just running out of memory. The way it runs out of memory is interesting though. Basically it makes so many calls to ModelicaExternal's ModelicaStrings_substring, with every call leaking memory of the returned substring. The whole process is like this

• omc sees a Modelica code calling Modelica.Utilities.Strings.substring
• omc tries to constant evaluate the function.
• omc loads libModelicaExternalC.dll based on the annotations.
• omc calls ModelicaStrings_substring() from libModelicaExternalC.dll using the libffi interface.
• ModelicaStrings_substring() calls ModelicaAllocateString() from our runtime library libOpenModelicaRuntimeC
• libOpenModelicaRuntimeC calls malloc from our garbage collector library libomcgc
• A new string is returned and omc finishes evaluating the call.
• omc unloads libModelicaExternalC.dll.
• The Garbage collector collects the dangling string memory.

Except that the last step seem to be never performed. The garbage collector does not collects the now unused memory. But it only happens on Windows.

See now, luckily for me, I have seen this before 🙂. I had moved the parallelization work from Windows to linux because the garbage collector kept segfaulting on me. We must have two instances of the garbage collector running.

There are some holes in my take on this but here it is:

• omc.exe (or rather libOpenModelicaCompiler.dll, the compiler itself) is linked to libOpenModelicaRuntimeC and libomcgc.

• libModelicaExternalC.dll is a DLL and is ALSO linked to libOpenModelicaRuntimeC and libomcgc which are both static libraries. So it has its own copy of the ModelicaAllocateString() function which calls its own separate garbage collector. It seems, when loaded, libModelicaExternalC.dll uses its own garbage collector and gets unloaded before having the chance to return the memory it allocated (??)

Why does this not happen on Linux? because libOpenModelicaRuntimeC and libomcgc both are compiled as shared libraries on linux. So there is always only one instance of the garbage collector. Compiling libOpenModelicaRuntimeC on Windows as a DLL ( (and making sure nothing else links to libomcgc) fixes the issue and the model does not segfault anymore. Compiling the Garbage collector (libomcgc) as a DLL is the final solution but it is ongoing due to some obstacles right now. I will try to finish it up and see what happens.

[fedetft]

Looks like you keep hitting the same problem of mixing static and dynamic linking, expecting the magic underneath to do more than it actually can do.

By linking the same library statically in multiple shared libraries, the functions will be duplicated, and even a slight misalignment between the two version will cause problems, and that is without considering that now the global variables will suddenly be duplicated and taking the address of functions (for function pointers) now returns a different pointer for the same function depending on which shared object you're calling.

I guess one of the possible outcomes of undefined behavior is that it works, but that's not a good argument to keep doing it and wasting debugging time.

Out of curiosity, what's stopping you from NOT LINKING STATICALLY any library that:
a) is used in more than one shared object, such as libomcgc?
b) is so common that you may anticipate it being used in external functions, such as libstdc++?

[mahge]

Looks like you keep hitting the same problem of mixing static and dynamic linking, expecting the magic underneath to do more than it actually can do.

We have to mix them of course. It is not all smooth sailing if all libraries are shared OR all libraries are static. They have their own use cases and their own benefits. Not to mention that we have a lot of 3rdParty libraries and we have different requirements even for our own libraries. Granted in this case these two specific libraries should have been shared. But from what I have seen someone probably took the easy way out and thought this would never happen. I am not sure.

The ground rule should be all position independent static libraries unless they need to be shared. That is what the new CMake build does.

By linking the same library statically in multiple shared libraries, the functions will be duplicated, and even a slight misalignment between the two version will cause problems, and that is without considering that now the global variables will suddenly be duplicated and taking the address of functions (for function pointers) now returns a different pointer for the same function depending on which shared object you're calling.

This can happen with DLLs and has happened before due to global variables in the garbage collector library. However, I do not think it is not what happened here. We have no mixing or one instance refereeing to the variables in the other instance if that makes sense. But you are right it is better to avoid it altogether.

Out of curiosity, what's stopping you from NOT LINKING STATICALLY any library that:
a) is used in more than one shared object, such as libomcgc?
b) is so common that you may anticipate it being used in external functions, such as libstdc++?

Good questions and I agree with what you are implying. I guess the answer is time. Most of this functionality was added a long time ago. At the time maybe there was no need to add more DLLs, maybe it was difficult to change after the fact. I can not say. OpenModelica has more than a 100 targets to be built. That is a lot to get right at once. Hindsight is 20/20 as they say.

If you ask me personally, I will point you to the commit message here: 23640ff

[fedetft]

I'll open another ticket for the issue with linking statically in the general case, but for the sake of solving the specific issue of externalmedia segfaulting, should it be possible to start doing the right thing and building libantlr4 dynamically, or at least building it statically, but without any -Wl,-Bstatic -lstdc++ -Wl,-Bdynamic nor -static-libstdc++?
That should be the simplest first thing to do, and it will solve the problem for Linux, then I suggest looking into linking libOpenModelicaRuntimeC and libomcgc dynamically on Windows, which should fix the other half of the issue.

[mahge]

#8954 should address the Linux std::cout related issue. I am cleaning up the fix for the Windows issue and I will push it in once it is ready.

[mahge]

The Windows issue should be fixed with #8961. We now build both libOpenModelicaCompiler and libomcgc as DLLs on Windows (they are already shared objects on Linux). Both the ExternalMedia ExternalMedia.Test.CoolProp.CO2.TestStatesSatSubcritical model and the MWE pass the checkModel successfully.

[fedetft]

Thanks!
Since I reverted back to the nightly builds, I guess I just need to wait for the next update for the problem to be fixed, right?

[mahge]

Yes unfortunately you will have to wait until tomorrow. Assuming nothing else goes wrong with the nightly builds.

If you really want to get going today you can try building OpenModelica from source using the CMake build. Since you are already familiar with CMake it should not be very difficult for you. If you need some more info you can find it here.

[casella]

Thanks @mahge :)