Fix bug of unittest (related to #507) #509

Clcanny · 2020-12-12T04:18:27Z

No description provided.

Clcanny · 2020-12-12T04:26:15Z

romainthomas · 2020-12-12T08:13:24Z

It looks like it still fails (https://travis-ci.org/github/lief-project/LIEF/jobs/749209301#L1798)

Clcanny · 2020-12-12T09:49:07Z

It looks like it still fails (https://travis-ci.org/github/lief-project/LIEF/jobs/749209301#L1798)

Can you tell me how to run travis ci on my local machine?

Clcanny · 2020-12-13T10:43:33Z

I am busy right now. This issue is still on working, please wait. Thanks.

Clcanny · 2020-12-16T17:31:42Z

The root cause is elf generated in liefproject/manylinux1_x86_64 image doesn't has .gnu.hash section.

In Debian GNU/Linux 9.9 (stretch):

# readelf --section-headers libfoo-modified.so | grep -E "Nr|.gnu.hash" -A1
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
--
  [ 2] .gnu.hash         GNU_HASH         000000000040a000  0000a000
       0000000000000054  0000000000000000   A       3     0     8

In docker container of liefproject/manylinux1_x86_64 image:

# readelf --section-headers libfoo-modified.so | grep -E "Nr|.gnu.hash" -A1
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align

romainthomas · 2020-12-17T07:51:21Z

Ack but please make sure to pass the tests locally before using the CI

Clcanny · 2020-12-21T04:58:37Z

Ack but please make sure to pass the tests locally before using the CI

Hello, sorry for the interruption. And this change is okay now.

romainthomas · 2020-12-21T15:07:41Z

Hello @Clcanny
I think that your PR is really useful nevertheless I disagree with your fix since it relies on a compiler flag and if the binary is compiled with -Wl,--hash-style=both it raises the following errors:

Symbol out-of-bound __gmon_start__
information of .dynsym section changes from 1 to 2
symndx of .gnu.hash section changes from 6 to 12
Symbol out-of-bound _ITM_registerTMCloneTable
symndx of .gnu.hash section changes from 6 to 12
Symbol out-of-bound _ITM_registerTMCloneTable
symndx of .gnu.hash section changes from 6 to 12

I had to revert the commit of your PR (see 51305e2) since it introduces breaking changes but I'll try to take a look during the holidays.

…both)

Clcanny · 2020-12-27T19:00:31Z

@romainthomas , sorry to disturb, wish you have a wonderful holiday.

I do the following things to improve my changes:

Handle SysV hash section correctly.
Let 3 python tests (SysV, GNU, both) pass.
Add a detailed test report.

I still can't promise that it won't break anything, but I will try my best to fix them if any bugs found.

Clcanny · 2020-12-27T19:09:16Z

Test report

Environment

Docker image name: liefproject/manylinux1_x86_64

Docker image id: 6efd2f2e96ed

Code

I feel hard to debug python test case, so I write cpp test case instead.

// foo.cpp
int uninitialized_var;
int initialized_var = 1;
int add(int a, int b) {
  return a + b;
}

// main.cpp
#include <iostream>
extern int uninitialized_var;
extern int initialized_var;
int add(int a, int b);
int main() {
  std::cout << add(uninitialized_var, initialized_var) << std::endl;
}

#include <LIEF/ELF.hpp>
#include <memory>
#include <vector>
int main() {
  std::unique_ptr<LIEF::ELF::Binary> binary_(
      LIEF::ELF::Parser::parse("libfoo.so"));
  std::vector<LIEF::ELF::Symbol> dynamic_symbols{
      binary_->dynamic_symbols().begin(), binary_->dynamic_symbols().end()};
  for (const auto& symbol : dynamic_symbols) {
    binary_->add_dynamic_symbol(symbol);
  }

  auto dynsym_section = binary_->get_section(".dynsym");
  binary_->extend(dynsym_section,
                  dynamic_symbols.size() * dynsym_section.entry_size());

  if (binary_->has_section(".hash")) {
    auto hash_section = binary_->get_section(".hash");
    binary_->extend(hash_section,
                    dynamic_symbols.size() * hash_section.entry_size());
  }

  binary_->write("libfoo-modified.so");
}

# Makefile
CC=/opt/devtools-6.3/bin/gcc
CXX=/opt/devtools-6.3/bin/g++
STDLIB_PATH=/opt/devtools-6.3/lib64
hash_style=both

compile : foo.cpp main.cpp merge.cpp
        $(CC) foo.cpp -shared -fPIC -Wl,--hash-style=$(hash_style) -o libfoo.so
        $(CXX) main.cpp -Wl,--hash-style=$(hash_style) -L$(PWD) -Wl,-rpath=$(PWD) -lfoo -o main
        $(CXX) merge.cpp -std=c++11 -I/usr/include/LIEF -L/usr/lib -lLIEF -o merge

run : compile libfoo.so main merge
        LD_LIBRARY_PATH=$(STDLIB_PATH) ./merge

test : run libfoo.so libfoo-modified.so main
        ./test.sh

#!/bin/bash

export origin_hash_section_start_addr="0x$(readelf --section-headers libfoo.so | grep ' .hash' | awk '{print $6}')"
export origin_hash_section_size="0x$(readelf --section-headers libfoo.so | grep ' .hash' -A1 | tail -n 1 | awk '{print $1}')"
export origin_nbucket="0x$(od --skip-bytes=$origin_hash_section_start_addr --read-bytes=8 --format=xI libfoo.so | head -n 1 | awk '{print $2}')"
export origin_nchain="0x$(od --skip-bytes=$origin_hash_section_start_addr --read-bytes=8 --format=xI libfoo.so | head -n 1 | awk '{print $3}')"

rm libfoo.so
chmod u+x libfoo-modified.so
mv libfoo-modified.so libfoo.so
if ./main | grep -q "1"
then
    echo "Run main successfully."
fi

export new_hash_section_start_addr="0x$(readelf --section-headers libfoo.so | grep ' .hash' | awk '{print $6}')"
export new_hash_section_size="0x$(readelf --section-headers libfoo.so | grep ' .hash' -A1 | tail -n 1 | awk '{print $1}')"
export new_nbucket="0x$(od --skip-bytes=$new_hash_section_start_addr --read-bytes=8 --format=xI libfoo.so | head -n 1 | awk '{print $2}')"
export new_nchain="0x$(od --skip-bytes=$new_hash_section_start_addr --read-bytes=8 --format=xI libfoo.so | head -n 1 | awk '{print $3}')"

if [[ $new_nbucket -eq $origin_nbucket ]]
then
    echo "new_bucket is equal to origin_nbucket."
fi

if [[ $new_nchain -eq $((origin_nchain * 2)) ]]
then
    echo "new_nchain is twice origin_nchain."
fi

if [[ $origin_hash_section_size -eq $((4 + 4 + origin_nbucket * 4 + origin_nchain * 4)) ]]
then
    echo "origin_hash_section_size is expected."
fi

if [[ $new_hash_section_size -eq $((4 + 4 + new_nbucket * 4 + new_nchain * 4)) ]]
then
    echo "new_hash_section_size is expected."
fi

Symbol orders are expected.

SysV

# make test hash_style=sysv
/opt/devtools-6.3/bin/gcc foo.cpp -shared -fPIC -Wl,--hash-style=sysv -o libfoo.so
/opt/devtools-6.3/bin/g++ main.cpp -Wl,--hash-style=sysv -L/root/test_compatibility_of_hash_table -Wl,-rpath=/root/test_compatibility_of_hash_table -lfoo -o main
# readelf --dyn-syms libfoo.so
Symbol table '.dynsym' contains 28 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     2: 00000000000027dc     0 FUNC    GLOBAL DEFAULT   11 _fini
     3: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
     5: 0000000000002678     0 FUNC    GLOBAL DEFAULT    7 _init
     6: 00000000002031a8     4 OBJECT  GLOBAL DEFAULT   20 initialized_var
     7: 00000000000027c8    20 FUNC    GLOBAL DEFAULT   10 _Z3addii
     8: 00000000002031b0     4 OBJECT  GLOBAL DEFAULT   21 uninitialized_var
     9: 00000000002031ac     0 NOTYPE  GLOBAL DEFAULT   21 __bss_start
    10: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
    11: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (2)
    12: 00000000002031ac     0 NOTYPE  GLOBAL DEFAULT   20 _edata
    13: 00000000002031b8     0 NOTYPE  GLOBAL DEFAULT   21 _end
    14: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
    15: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
    16: 00000000000027dc     0 FUNC    GLOBAL DEFAULT   11 _fini
    17: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
    18: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
    19: 0000000000002678     0 FUNC    GLOBAL DEFAULT    7 _init
    20: 00000000002031a8     4 OBJECT  GLOBAL DEFAULT   20 initialized_var
    21: 00000000000027c8    20 FUNC    GLOBAL DEFAULT   10 _Z3addii
    22: 00000000002031b0     4 OBJECT  GLOBAL DEFAULT   21 uninitialized_var
    23: 00000000002031ac     0 NOTYPE  GLOBAL DEFAULT   21 __bss_start
    24: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
    25: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize
    26: 00000000002031ac     0 NOTYPE  GLOBAL DEFAULT   20 _edata
    27: 00000000002031b8     0 NOTYPE  GLOBAL DEFAULT   21 _end

They just keep the same order, I don't sort them. It is expetced.

GNU

# make test hash_style=gnu
/opt/devtools-6.3/bin/gcc foo.cpp -shared -fPIC -Wl,--hash-style=gnu -o libfoo.so
/opt/devtools-6.3/bin/g++ main.cpp -Wl,--hash-style=gnu -L/root/test_compatibility_of_hash_table -Wl,-rpath=/root/test_compatibility_of_hash_table -lfoo -o main
# readelf --dyn-syms libfoo.so
Symbol table '.dynsym' contains 28 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     2: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     3: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
     5: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
     6: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (2)
     7: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     8: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
     9: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
    10: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
    11: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize
    12: 0000000000204178     4 OBJECT  GLOBAL DEFAULT   21 uninitialized_var
    13: 0000000000204174     0 NOTYPE  GLOBAL DEFAULT   20 _edata
    14: 0000000000204180     0 NOTYPE  GLOBAL DEFAULT   21 _end
    15: 0000000000204178     4 OBJECT  GLOBAL DEFAULT   21 uninitialized_var
    16: 0000000000204174     0 NOTYPE  GLOBAL DEFAULT   20 _edata
    17: 0000000000204180     0 NOTYPE  GLOBAL DEFAULT   21 _end
    18: 0000000000003638     0 FUNC    GLOBAL DEFAULT    7 _init
    19: 0000000000204174     0 NOTYPE  GLOBAL DEFAULT   21 __bss_start
    20: 0000000000003638     0 FUNC    GLOBAL DEFAULT    7 _init
    21: 0000000000204174     0 NOTYPE  GLOBAL DEFAULT   21 __bss_start
    22: 00000000000037a4     0 FUNC    GLOBAL DEFAULT   11 _fini
    23: 0000000000204170     4 OBJECT  GLOBAL DEFAULT   20 initialized_var
    24: 0000000000003790    20 FUNC    GLOBAL DEFAULT   10 _Z3addii
    25: 00000000000037a4     0 FUNC    GLOBAL DEFAULT   11 _fini
    26: 0000000000204170     4 OBJECT  GLOBAL DEFAULT   20 initialized_var
    27: 0000000000003790    20 FUNC    GLOBAL DEFAULT   10 _Z3addii

They are sorted, and it is expected.

Both

# make test hash_style=both
/opt/devtools-6.3/bin/gcc foo.cpp -shared -fPIC -Wl,--hash-style=both -o libfoo.so
/opt/devtools-6.3/bin/g++ main.cpp -Wl,--hash-style=both -L/root/test_compatibility_of_hash_table -Wl,-rpath=/root/test_compatibility_of_hash_table -lfoo -o main
# readelf --dyn-syms libfoo.so
Symbol table '.dynsym' contains 28 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     2: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     3: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
     5: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
     6: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (2)
     7: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     8: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
     9: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
    10: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
    11: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize
    12: 00000000002041b0     4 OBJECT  GLOBAL DEFAULT   22 uninitialized_var
    13: 00000000002041ac     0 NOTYPE  GLOBAL DEFAULT   21 _edata
    14: 00000000002041b8     0 NOTYPE  GLOBAL DEFAULT   22 _end
    15: 00000000002041b0     4 OBJECT  GLOBAL DEFAULT   22 uninitialized_var
    16: 00000000002041ac     0 NOTYPE  GLOBAL DEFAULT   21 _edata
    17: 00000000002041b8     0 NOTYPE  GLOBAL DEFAULT   22 _end
    18: 00000000000036c0     0 FUNC    GLOBAL DEFAULT    8 _init
    19: 00000000002041ac     0 NOTYPE  GLOBAL DEFAULT   22 __bss_start
    20: 00000000000036c0     0 FUNC    GLOBAL DEFAULT    8 _init
    21: 00000000002041ac     0 NOTYPE  GLOBAL DEFAULT   22 __bss_start
    22: 000000000000382c     0 FUNC    GLOBAL DEFAULT   12 _fini
    23: 00000000002041a8     4 OBJECT  GLOBAL DEFAULT   21 initialized_var
    24: 0000000000003818    20 FUNC    GLOBAL DEFAULT   11 _Z3addii
    25: 000000000000382c     0 FUNC    GLOBAL DEFAULT   12 _fini
    26: 00000000002041a8     4 OBJECT  GLOBAL DEFAULT   21 initialized_var
    27: 0000000000003818    20 FUNC    GLOBAL DEFAULT   11 _Z3addii

They are sorted, and it is expected.

Sizes of .hash section are expected.

When you run make test hash_style=sysv and make test hash_style=both, they will say:

Run main successfully.
new_bucket is equal to origin_nbucket.
new_nchain is twice origin_nchain.
origin_hash_section_size is expected.
new_hash_section_size is expected.

src/ELF/Builder.tcc

tests/elf/test_dynamic.py

Clcanny · 2021-01-27T05:21:57Z

@romainthomas Hello, I notice that the new version is released. Is it a good time to merge this pull request? Thank you.

romainthomas · 2021-01-27T05:28:23Z

Yes I didn't forget, I'll do that in the next couple of weeks

romainthomas · 2021-01-31T10:10:50Z

done with 3d0da01

Clcanny force-pushed the charles-fix-ut-bug-507 branch from f4aad63 to 71941f8 Compare December 16, 2020 17:27

Clcanny force-pushed the charles-fix-ut-bug-507 branch 3 times, most recently from 4bc7865 to 1977479 Compare December 17, 2020 07:16

Clcanny force-pushed the charles-fix-ut-bug-507 branch from 1977479 to 7b36d86 Compare December 21, 2020 03:25

Clcanny added 2 commits December 23, 2020 19:58

Fix bug of unittest (related to lief-project#507)

4ddd109

Let sort_dynamic_symbols works for different hash styles (sysv, gnu, …

7edb1db

…both)

Clcanny force-pushed the charles-fix-ut-bug-507 branch from 7b36d86 to 7edb1db Compare December 23, 2020 12:15

Resolve conflits after rebase

3ae0181

Clcanny marked this pull request as draft December 23, 2020 17:27

Extend .hash section

28a1b3d

Clcanny marked this pull request as ready for review December 27, 2020 19:26

Improve warning

1492638

romainthomas reviewed Jan 4, 2021

View reviewed changes

src/ELF/Builder.tcc Show resolved Hide resolved

romainthomas reviewed Jan 4, 2021

View reviewed changes

tests/elf/test_dynamic.py Show resolved Hide resolved

romainthomas closed this Jan 31, 2021

jheydebrand mentioned this pull request May 27, 2021

.dynsym not sorted by symbol visibility when using sysv hash style #582

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix bug of unittest (related to #507) #509

Fix bug of unittest (related to #507) #509

Clcanny commented Dec 12, 2020

Clcanny commented Dec 12, 2020

romainthomas commented Dec 12, 2020

Clcanny commented Dec 12, 2020

Clcanny commented Dec 13, 2020

Clcanny commented Dec 16, 2020

romainthomas commented Dec 17, 2020

Clcanny commented Dec 21, 2020

romainthomas commented Dec 21, 2020

Clcanny commented Dec 27, 2020 •

edited

Loading

Clcanny commented Dec 27, 2020 •

edited

Loading

Clcanny commented Jan 27, 2021

romainthomas commented Jan 27, 2021

romainthomas commented Jan 31, 2021

Fix bug of unittest (related to #507) #509

Fix bug of unittest (related to #507) #509

Conversation

Clcanny commented Dec 12, 2020

Clcanny commented Dec 12, 2020

romainthomas commented Dec 12, 2020

Clcanny commented Dec 12, 2020

Clcanny commented Dec 13, 2020

Clcanny commented Dec 16, 2020

romainthomas commented Dec 17, 2020

Clcanny commented Dec 21, 2020

romainthomas commented Dec 21, 2020

Clcanny commented Dec 27, 2020 • edited Loading

Clcanny commented Dec 27, 2020 • edited Loading

Test report

Environment

Code

Symbol orders are expected.

SysV

GNU

Both

Sizes of .hash section are expected.

Clcanny commented Jan 27, 2021

romainthomas commented Jan 27, 2021

romainthomas commented Jan 31, 2021

Clcanny commented Dec 27, 2020 •

edited

Loading

Clcanny commented Dec 27, 2020 •

edited

Loading