Inclusion of nlohmann/json.hpp causes OS/ABI to change on Linux

[jongough]

• What is the issue you have?
  Trying to incorporate a json schema validator using nlohmann/json.hpp into C++ code causes the linux gcc compiler/linker to change the OSABI from UNIX - System V (ELF OS/ABI value 0) to UNIX - GNU (ELF OS/ABI value 3). This makes it impossible to run with existing programs as it fails the compliance test at runtime.

• Please describe the steps to reproduce the issue. Can you provide a small but working code example?
  A simple program called testOSABI.cpp:

    //#include "nlohmann/json.hpp"
    int main (){
        return 0;
    }

If compiled with:
gcc testOSABI.cpp -o testOSABI.o -Inlohmann_json/single_include
Then use readelf
readelf -h testOSABI.o
will give:

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x4f0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          6376 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         28
  Section header string table index: 27

If the program has the #include statement uncommented and the same process followed the readelf will give:

  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x520
  Start of program headers:          64 (bytes into file)
  Start of section headers:          7032 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         29
  Section header string table index: 28

• What is the expected behaviour?
  The two outputs should be the same for the OS/ABI value

• And what is the actual behaviour instead?
  The two OS/ABI values are different. This will stop a program with json.hpp included from interacting successfully with one that does not have json.hpp included.

• Which compiler and operating system are you using? Is it a supported compiler?
  gcc 7.3.0 on Ubuntu based Linux Mint 19 with kernel 4.18.0-13-generic

• Did you use a released version of the library or the version from the develop branch?
  Release

[nlohmann]

Thanks for reporting! I have no idea how to proceed here.

[jongough]

I have looked around to see if there is anyway to set the OS/ABI with the compiler/linker options, but it seems to be poorly documented. I have tried some suggestion, i.e. -fno-jump-tables, but none of them worked.
The only thing that I can think of is to try cutting down the json.hpp file to see what capability you are using that changes the ELF flavour. I have also tried compiling my project for windows and had a similar result i.e. incompatible DLL is generated and causes a crash.

The code I am trying to integrate your json.hpp with is quite large with multiple modules generated by many people. In Linux the main program and all the loadable libraries have System V ABI, so I need to be able to generate my loadable with json.hpp the same.

[nickaein]

To track down the issue, we can use nm tool to see the symbols in the object:

nm --demangle testOSABI.o

Comparing its output to the case when jhon.hpp is not included, there are two symbols with u type which we hadn't before:

0000000000491f15 u nlohmann::detail::static_const<nlohmann::detail::to_json_fn>::value
0000000000491f14 u nlohmann::detail::static_const<nlohmann::detail::from_json_fn>::value

For u symbols nm man page says (emphasis mine):

The symbol is a unique global symbol. This is a GNU extension to the standard set of ELF symbol bindings. For such a symbol the dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use.

In the source code, to_json_fn and from_json_fn are the only types that are referenced inside a static_const template which in turn defined as:

template<typename T>
struct static_const
{
    static constexpr T value{};
};

template<typename T>
constexpr T static_const<T>::value;

I suspect the static constexpr translate as unique global variable to the linker so it uses unique global symbol for these types. The root cause could be new revisions to the standard abbout static variables, their initialization their thread-safety. I might be wrong on this though.

However, I'm afraid this is not limited only to from_json and to_json.

Take the following code for instance:

#include <string>

int main()
{
    std::string str;

    size_t pos = str.find("Hello");
    if(pos == std::string::npos)
        return 0;

    return 1;
}

Compiling this code with GCC 7.3 also lead to a binary with header OS/ABI: UNIX - GNU. The nm output confirms it too as there is a std::string::npos symbol with type u. This means that it is probably defined as static constexpr or something similar to that in the std::string source code.

This is not always true. A better code sample is posted in my next comment: #1410 (comment)

[jongough]

In my program with json.hpp included I get the following " u " values

00000000005d10c8 u guard variable for std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, false, true>::operator()(char) const::__nul
00000000005d10d8 u guard variable for std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, true, false>::operator()(char) const::__nul
00000000005d10e8 u guard variable for std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, true, true>::operator()(char) const::__nul
00000000005b77e0 u std::__cxx11::regex_traits<char>::lookup_classname<char const*>(char const*, char const*, bool) const::__classnames
00000000005cfcc0 u std::__cxx11::regex_traits<char>::lookup_collatename<char const*>(char const*, char const*) const::__collatenames
00000000005d10b0 u std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, false, false>::operator()(char) const::__nul
00000000005d10c0 u std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, false, true>::operator()(char) const::__nul
00000000005d10d0 u std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, true, false>::operator()(char) const::__nul
00000000005d10e0 u std::__detail::_AnyMatcher<std::__cxx11::regex_traits<char>, false, true, true>::operator()(char) const::__nul
000000000032a178 u std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, false>::_M_is_word(char) const::__s
000000000032a17a u std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_M_is_word(char) const::__s

Is there any ability to change this so that GNU extensions are not used? The code my module is interacting with is UNIX - System V and cannot be changed to UNIX - GNU. The code is also used on osX and Windows, and I know windows gets a 'similar' result, although the error codes are different.

Regards
Jon

[nickaein]

In my program with json.hpp included I get the following " u " values

The "u" symbols present in the object may differ based on your usage. That's because the linker can removed unused symbols from the final executable.

It is interesting that all "u" symbols showing up for you is from std namespace though. It's weird that there are no "u" symbols from nlohmann json namespace as in my experience a couple of them should up. Maybe linker removed it's unused symbols?

Is there any ability to change this so that GNU extensions are not used?

I am no expert on this but and it seems not easy. Here are some posts discussing this issue:

https://stackoverflow.com/questions/11931420
https://gcc.gnu.org/ml/gcc-help/2013-01/msg00008.html
https://cygwin.com/ml/binutils/2011-10/msg00276.html

There are probably various cases that such symbol shows up in the binary object. The previous sample code I posted at #1410 (comment) doesn't always work.

The following code (which is very simple singleton pattern) seems trigger "u" symbol generation too.
This happens even on c++98 standard (g++ -std=c++98 -O3 main.cpp -o main.o):

class singleton {
public:
    static int& getInstance()
    {
        static int number = 1;

        return number;
    }
};

int main()
{
    int& ref = singleton::getInstance();

    return ref;
}

[FrancoisChabot]

This can be fixed by simply wrapping static_const in an annonymous namespace, thus preventing the symbols from being exported altogether (which should be fine for a constexpr value like this):

namespace {
template<typename T>
struct static_const
{
    static constexpr T value{};
};

template<typename T>
constexpr T static_const<T>::value;
}
}  // namespace detail
}  // namespace nlohmann

Whether this should be done for the library is potentially contentious since it's an import from ranges-V3 and consistency might be desirable. However, it's a change you can make to your own copy of json.hpp very easily.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jongough]

I was away so I could not do a test before the bot auto closed this issue.

I have tried this, but it still shows the same " u " strings. So there must be something else in the json.hpp which is triggering this behaviour.

[nlohmann]

Should I reopen? Does anyone has an idea?

[jongough]

It needs someone to look at it as just including json.hpp should leave the 'flavour' of the environment as it was, i.e. it should add no 'special' processing. If this is not going to be the case then it makes it very difficult for developers to use this code to assist with json processing.

[nlohmann]

I do not know what you mean with "special processing" or "flavor of the environment". :-/

[jongough]

Sorry, Australian colloquialisms!
The use of json.hpp should not change the OSABI type and should work with whatever OSABI type the project is using. i.e. the json.hpp should not use any coding that requires the compilers to use a specific OSABI type to make it work. I know this may be difficult but as source code that can be added to any project to enable the proper working of json it is important.

I want to use this code to enable the use of json schemas for validating messages passing information between plugins within a program and between instances of the program running on different machines. I suspect that when this has been done it will then be used to pass information to other programs/projects as well. Due to the nature of the project I am working on I cannot change the OSABI, which is currently "UNIX - System V", i.e. generic, as the project is built for Linux, Windows, osX and Raspberry PI and is worked on, like yours, by an international team.

I tried the fix that FrancoisChabot suggested, but it made no difference, the OSABI still changed. I am not familiar enough with the use of namespaces, etc., to workout what is going on or where to make changes to fix the issue.

I gave an example program that demonstrates the problem when I opened this issue. If you want to look at my plugin (it will compile and link independent of the main program, but does require wxWidgets 3.1.2) you will find it here: https://github.com/jongough/ocpn_draw_pi . To use the json.hpp you will need to set 'OD_JSON_SCHEMA_VALIDATOR' to true in the CMakeCache.txt file. You will see the same output as I posted on the 7th Jan when issuing this command:
nm --demangle libocpn_draw_pi.so |grep " u "

[nlohmann]

I am not at all familiar in these issues. I hope someone who is can support here.

[nlohmann]

Maybe relates: https://www.acodersjourney.com/20-abi-breaking-changes/

[jongough]

That certainly looks close, particularly from item 8 onwards. I think you may be doing 'stuff' with const variables that could be causing the issue, certainly FrancoisChabot seemed to be suggesting changes to try and hide this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jongough]

Is there any work progressing on this issue? It is being marked as stale, but, to me, it appears still to be an issue.

[jongough]

It looks like this issue is not solvable, so I probably wont be able to use this work in my project.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jongough]

I have got the schema checking working but I have to hand change the OS/ABI back to the default. I have no idea what is causing the issue, but on linux changing the OS/ABI seems to work. I have not tried it yet on windows to see if it is a problem. But it would be very helpful if you could find out why your code is doing this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.