New instruction decoding problem in master branch

[mxz297]

In the xhpl binary provided from Rice, objdump shows the following instruction at the given address:

9aeef7: 49 63 fa movslq %r10d,%rdi

Current master branch decodes the instruction as

9aeef7 : arpl R10W, DI

[jdetter]

The issue is, arpl is a oneByteMap instruction that becomes movslq when it is REX prefixed. We don't have any way of differentiating between REX and non REX prefixed instructions.

[mxz297]

Current fix seems to lead to an assertion failure at
4591c1: 48 63 ff movslq %edi,%rdi

[jdetter]

Could you post that stacktrace of your issue? I'm getting an assert in instructionAPI and I think your assert was in common.

[mxz297]

Here it is.

#0  0x00007ffff5b8e5d7 in raise () from /lib64/libc.so.6
#1  0x00007ffff5b8fcc8 in abort () from /lib64/libc.so.6
#2  0x00007ffff5b87546 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff5b875f2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff67a835f in NS_x86::ia32_decode (capa=capa@entry=1, addr=<optimized out>, instruct=...) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/common/src/arch-x86.C:8703
#5  0x00007ffff7643817 in Dyninst::InstructionAPI::InstructionDecoder_x86::doIA32Decode (this=0x13e1790, b=...)
    at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/instructionAPI/src/InstructionDecoder-x86.C:1714
#6  0x00007ffff7643d59 in Dyninst::InstructionAPI::InstructionDecoder_x86::decodeOpcode (this=<optimized out>, b=...)
    at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/instructionAPI/src/InstructionDecoder-x86.C:1772
#7  0x00007ffff77542b4 in Dyninst::InstructionAPI::InstructionDecoderImpl::decode (this=0x13e1790, b=...) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/instructionAPI/src/InstructionDecoderImpl.C:56
#8  0x00007ffff763de49 in Dyninst::InstructionAPI::InstructionDecoder_x86::decode (this=<optimized out>, b=...)
    at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/instructionAPI/src/InstructionDecoder-x86.C:1836
#9  0x00007ffff763dc5c in Dyninst::InstructionAPI::InstructionDecoder::decode (this=this@entry=0x7fffffffd240) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/instructionAPI/src/InstructionDecoder.C:65
#10 0x00007ffff7b662ac in Dyninst::InsnAdapter::IA_IAPI::isStackFramePreamble (this=this@entry=0x7fffffffd310) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/parseAPI/src/IA_x86.C:415
#11 0x00007ffff79c5c1c in Dyninst::ParseAPI::Parser::init_frame (this=this@entry=0x111b7c0, frame=...) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/parseAPI/src/Parser.C:781
#12 0x00007ffff79c7534 in Dyninst::ParseAPI::Parser::parse_vanilla (this=this@entry=0x111b7c0) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/parseAPI/src/Parser.C:273
#13 0x00007ffff79c789d in Dyninst::ParseAPI::Parser::parse (this=0x111b7c0) at /p/paradyn/development/xmeng/dyninstapi/dyninst_development/dyninst/parseAPI/src/Parser.C:157
#14 0x0000000000400f11 in main (argc=<optimized out>, argv=<optimized out>) at indirect.cpp:74

[jdetter]

I think I figured it out, there was a table indexing issue where the decoder went passed the end of the table (oops).

[mxz297]

We are moving on the right direction, but not there yet. Objdump shows:

9b4da2: 4c 63 cb movslq %ebx,%r9

I got

9b4da2 : movslq RBX, R9

The operand order and the operand size (ebx vs rbx) seem different.

[jdetter]

You're right, I added them in AT&T syntax order and dyninst was uses Intel order.

[jdetter]

I don't think there will be any semantics for that instruction because I just added it to dyninst, is that okay?

[jdetter]

After adding that change, I am now getting an assert in the x86 instruction semantics:

#0  0x00007ffff4a51a28 in raise () from /usr/lib64/libc.so.6
#1  0x00007ffff4a5362a in abort () from /usr/lib64/libc.so.6
#2  0x00007ffff4a4a227 in __assert_fail_base () from /usr/lib64/libc.so.6
#3  0x00007ffff4a4a2d2 in __assert_fail () from /usr/lib64/libc.so.6
#4  0x00007ffff7a241ad in X86_64InstructionSemantics<Dyninst::DataflowAPI::SymEvalPolicy_64, Dyninst::DataflowAPI::Handle>::read8 (this=0x7fffffffb230, e=0x3ca2860)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/../rose/x86_64InstructionSemantics.h:186
#5  0x00007ffff79cecbe in X86_64InstructionSemantics<Dyninst::DataflowAPI::SymEvalPolicy_64, Dyninst::DataflowAPI::Handle>::translate (this=0x7fffffffb230, insn=0x39d1200)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/../rose/x86_64InstructionSemantics.h:753
#6  0x00007ffff799d375 in X86_64InstructionSemantics<Dyninst::DataflowAPI::SymEvalPolicy_64, Dyninst::DataflowAPI::Handle>::processInstruction (this=0x7fffffffb230, insn=0x39d1200)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/../rose/x86_64InstructionSemantics.h:2732
#7  0x00007ffff7999e8d in Dyninst::DataflowAPI::SymbolicExpansion::expandX86_64 (
    rose_insn=0x39d1200, policy=...)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/SymbolicExpansion.C:60
#8  0x00007ffff7a54c07 in Dyninst::DataflowAPI::SymEval::expandInsn (insn=...,
    addr=9390605, res=std::map with 1 elements = {...})
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/SymEval.C:455
#9  0x00007ffff7a52d69 in Dyninst::DataflowAPI::SymEval::expand (
    res=std::map with 1 elements = {...}, failedInsns=std::set with 0 elements,
    applyVisitors=false) at /home/detter/Workspace/xmeng-test/dataflowAPI/src/SymEval.C:93
#10 0x00007ffff7a52b94 in Dyninst::DataflowAPI::SymEval::expand (assignment=...,
    applyVisitors=false) at /home/detter/Workspace/xmeng-test/dataflowAPI/src/SymEval.C:74
#11 0x00007ffff78a24c3 in JumpTablePred::ExpandAssignment (this=0x7fffffffcaf0, assign=...)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/JumpTablePred.C:402
#12 0x00007ffff78a0b8d in JumpTablePred::addNodeCallback (this=0x7fffffffcaf0, ap=...,
    visitedEdges=std::set with 0 elements)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/JumpTablePred.C:193
#13 0x00007ffff793f511 in Dyninst::Slicer::updateAndLink (this=0x7fffffffcbd0, g=...,
    dir=Dyninst::Slicer::backward, cand=..., cache=..., p=...)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:398
#14 0x00007ffff793e8a6 in Dyninst::Slicer::sliceInternalAux (this=0x7fffffffcbd0, g=...,
    dir=Dyninst::Slicer::backward, p=..., cand=..., skip=false,
    visited=std::map with 4 elements = {...},
    singleCache=std::map with 5 elements = {...}, cache=std::map with 0 elements)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:231
#15 0x00007ffff793ee61 in Dyninst::Slicer::sliceInternalAux (this=0x7fffffffcbd0, g=...,
    dir=Dyninst::Slicer::backward, p=..., cand=..., skip=false,
    visited=std::map with 4 elements = {...},
    singleCache=std::map with 5 elements = {...}, cache=std::map with 0 elements)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:307
#16 0x00007ffff793ee61 in Dyninst::Slicer::sliceInternalAux (this=0x7fffffffcbd0, g=...,
    dir=Dyninst::Slicer::backward, p=..., cand=..., skip=false,
    visited=std::map with 4 elements = {...},
    singleCache=std::map with 5 elements = {...}, cache=std::map with 0 elements)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:307
#17 0x00007ffff793ee61 in Dyninst::Slicer::sliceInternalAux (this=0x7fffffffcbd0, g=...,
    dir=Dyninst::Slicer::backward, p=..., cand=..., skip=false,
    visited=std::map with 4 elements = {...},
    singleCache=std::map with 5 elements = {...}, cache=std::map with 0 elements)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:307
#18 0x00007ffff793ee61 in Dyninst::Slicer::sliceInternalAux (this=0x7fffffffcbd0, g=...,
    dir=Dyninst::Slicer::backward, p=..., cand=..., skip=true,
    visited=std::map with 4 elements = {...},
    singleCache=std::map with 5 elements = {...}, cache=std::map with 0 elements)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:307
#19 0x00007ffff793e56b in Dyninst::Slicer::sliceInternal (this=0x7fffffffcbd0,
    dir=Dyninst::Slicer::backward, p=...)
    at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:195
#20 0x00007ffff7943bc8 in Dyninst::Slicer::backwardSlice (this=0x7fffffffcbd0,
    predicates=...) at /home/detter/Workspace/xmeng-test/dataflowAPI/src/slicing.C:1394
#21 0x00007ffff78d0869 in IndirectControlFlowAnalyzer::NewJumpTableAnalysis (
    this=0x7fffffffd080, outEdges=std::vector of length 0, capacity 0)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/IndirectAnalyzer.C:45
#22 0x00007ffff78823ed in Dyninst::InsnAdapter::IA_x86Details::parseJumpTable (
    this=0x2d8c8b0, currFunc=0x1288f90, currBlk=0x39c3930,
    outEdges=std::vector of length 0, capacity 0)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/IA_x86Details.C:1002
#23 0x00007ffff78744bb in Dyninst::InsnAdapter::IA_IAPI::parseJumpTable (this=0x3e41450,
    currFunc=0x1288f90, currBlk=0x39c3930, outEdges=std::vector of length 0, capacity 0)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/IA_IAPI.C:958
#24 0x00007ffff7872c29 in Dyninst::InsnAdapter::IA_IAPI::getNewEdges (this=0x3e41450,
    outEdges=std::vector of length 0, capacity 0, context=0x1288f90, currBlk=0x39c3930,
    num_insns=605, plt_entries=0xe6a0d8, knownTargets=std::set with 174 elements = {...})
    at /home/detter/Workspace/xmeng-test/parseAPI/src/IA_IAPI.C:681
#25 0x00007ffff7803389 in Dyninst::ParseAPI::Parser::ProcessCFInsn (this=0xe69fe0,
    frame=..., cur=0x39c3930, ah=...)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/ParserDetails.C:425
#26 0x00007ffff781146d in Dyninst::ParseAPI::Parser::parse_frame (this=0xe69fe0,
    frame=..., recursive=true)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/Parser.C:1119
#27 0x00007ffff780dc11 in Dyninst::ParseAPI::Parser::parse_frames (this=0xe69fe0,
    work=std::vector of length 3, capacity 8 = {...}, recursive=true)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/Parser.C:406
#28 0x00007ffff780d03c in Dyninst::ParseAPI::Parser::parse_at (this=0xe69fe0, region=
    0xebaf80, target=9368576, recursive=true, src=Dyninst::ParseAPI::ONDEMAND)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/Parser.C:211
#29 0x00007ffff780d21e in Dyninst::ParseAPI::Parser::parse_at (this=0xe69fe0,
    target=9368576, recursive=true, src=Dyninst::ParseAPI::ONDEMAND)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/Parser.C:241
#30 0x00007ffff784d00c in Dyninst::ParseAPI::CodeObject::parse (this=0xe5fef0,
    target=9368576, recursive=true)
    at /home/detter/Workspace/xmeng-test/parseAPI/src/CodeObject.C:165
#31 0x000000000040251a in main (argc=2, argv=0x7fffffffe568) at src/main.cpp:91

assert message:

dyn_dump: /home/detter/Workspace/xmeng-test/dataflowAPI/src/../rose/x86_64InstructionSemantics.h:186: WordType<8ul> X86_64InstructionSemantics<Policy, WordType>::read8(SgAsmExpression*) [with Policy = Dyninst::DataflowAPI::SymEvalPolicy_64; WordType = Dyninst::DataflowAPI::Handle]: Assertion `!"Bad position in register"' failed.