Import all changes from the Ubuntu fork

[morphis]

This import nearly all changes we did in Ubuntu on top of a libhybris release from the end of last year. There are quite some differences but a lot of features added.

• Android 6 support
• Arm64 support
• Dynamic linker support
• A lot enablement changes

I didn't went through the list of other changes done in upstream but I feel the easiest is the apply any needed ones again on top of this giant merge proposal (if still needed).

[morphis]

Fixed.

[krnlyng]

42e27f9 is kinda hacky but much more nice than what i have done locally 👍

3ab2ca5 needs to be thought about, can we build everything into libhybris-common and still select the linkers at runtime? probably would require us to rename most linker functions, or is there a saner way?

7ae2b39 can this be made with env variable HYBRIS_LINKER_STDOUT=1 like it was before?

is the packaging really necessary in this repository? i kind of liked the idea of having a generic repository + a packaging repository with a submodule.

can we have the tracing functionality as a debug option without adding a special hook for every symbol?
i am thinking about something like this (simplified and pseudocode):

compile time solution:

#ifdef DEBUG
... all the indirect hooks useful for tracing ...
#endif

#ifdef DEBUG
#define HOOK_DIRECT(symbol) {#symbol, _hybris_hook_##symbol}
#else
#define HOOK_DIRECT(symbol) {#symbol, symbol},
#endif

#define HOOK(symbol) {#symbol, _hybris_hook##symbol}
#define HOOK_TO(symbol, hook) {#symbol, hook}

struct _hook hooks[] = {
...
    HOOK_DIRECT(strcmp), // for functions that can be hooked directly
    HOOK(fgets), // for functions that cannot be directly hooked
    HOOK_TO(__progname, something_other_than__hybris_hook), // for variables
...
};

OR at runtime:

struct _hook hooks[] = {
...
    {"strcmp", strcmp},
...
};
struct _hook debug_hooks[] = {
...
    {"strcmp", _hybris_hook_strcmp},
...
};

void *get_hooked_symbol(const char *symbol)
{
    if env variable HYBRIS_TRACE is set
        return symbol from debug_hooks;
    else
        return symbol from hooks;
}

thanks for adding the PR!

[morphis]

For 3ab2ca5 I agree that this can be done better but we didn't found a good think when adding the dynamic linker in time to get things working well. With the indirect stuff libGLESv2 was crashing at multiple places. Everything else should be fine and working. Also could we make the dynamic linker an optional feature but that would make the test scenarios quite more complex.

For 7ae2b39 , sure we can add those env variables back to the mm linker. We didn't had the time yet to do that. Patches welcome :-)

For the hook/trace thing I like what you propose to do at runtime but actually this should be covered with some nice macros too.

[krnlyng]

@morphis seems like _hybris_hook_scandir and my_scandir are duplicates of each other
same for scandirat
alphasort, versionsort seem to be hooked twice too (there are more duplicate hooks)

EDIT: can you also provide a testcase for the glesv2 issue? i cannot seem to reproduce any crashes with the indirect functions enabled, this is what i am using now: https://github.com/krnlyng/libhybris/tree/ubuntu

[ghosalmartin]

With this version of the linker it seemed to crash inside is_gnu_hash for EGL libs.

[bhush9]

backtrace of test_hwcomposer for record

#0  soinfo::is_gnu_hash (this=this@entry=0x6c2f7369) at linker.cpp:2385
#1  0xf73bb154 in soinfo::find_symbol_by_name (this=this@entry=0x6c2f7369, symbol_name=..., vi=vi@entry=0x0, symbol=symbol@entry=0xfffef270) at linker.cpp:491
#2  0xf73bb4c0 in <lambda(soinfo*)>::operator() (current_soinfo=0x6c2f7369, __closure=0xfffef22c) at linker.cpp:938
#3  walk_dependencies_tree<dlsym_handle_lookup(soinfo*, soinfo*, soinfo**, SymbolName&)::<lambda(soinfo*)> >(soinfo **, <lambda(soinfo*)>, size_t) ( root_soinfos=root_soinfos@entry=0xfffef264, action=..., root_soinfos_size=1) at linker.cpp:912
#4  0xf73bbcce in dlsym_handle_lookup (symbol_name=..., found=0xfffef2b4, skip_until=0x0, root=0x6c2f7369) at linker.cpp:949
#5  dlsym_handle_lookup (si=si@entry=0x6c2f7369, found=found@entry=0xfffef2b4, name=name@entry=0xf7356695 "eglCreateStreamKHR") at linker.cpp:969
#6  0xf73b9596 in android_dlsym (handle=0x6c2f7369, symbol=0xf7356695 "eglCreateStreamKHR") at dlfcn.cpp:127
#7  0xf6696628 in ?? ()

[ghosalmartin]

@bhush9 Did the one from my fork still have the same issue?

[bhush9]

I am not super sure.. but I can retry in sometime.

EDIT: I confused different issue with this one, I still have this linker crash with ubuntu's mm linker and had separate issue with fork and not is_gnu_hash crash.

[krnlyng]

@morphis we're still investigating the is_gnu_hash issue, but i have added some commits on top of this: https://github.com/krnlyng/libhybris/tree/ubuntu it works well on my device and solves a segfault on exit of all processes (7891cf2). please take a look.

EDIT: @bhush9 this has been resolved by ghosalmartin, try: https://github.com/krnlyng/libhybris/tree/ubuntu

[krnlyng]

could it be that b4c82eb is the cause for the random crashes inside GLESv2, at least on arm? GLES2_IDLOAD does not handle floating point arguments afaik (and even cannot handle those)

[morphis]

@krnlyng You need to be careful with the linker you select on your platform. The hash scheme was changed with the Android 5 linker and if there are old binaries in your system things may conflict.

I didn't looked into the details of your changes yet but I would propose we merge this PR and then get your changes on top of this one as followup PRs. How does that sound?

[krnlyng]

@morphis we have selected the correct linker, the issue was due to a pthread_getspecific issue where it loaded a key which was not initialized which caused all kinds of confusion within the blobs when dynamically loading other libraries

we're still looking into getting this to work on our devices, hybris-13 adaptation works fine now with some additional changes but we want to have it work on older adaptations too to make the transition as smooth (as in be ready with our pull requests as soon as this has been merged) as possible, also we're still unsure about the packaging inside this repository since it bloats the history unnecessarily with commits that are either not useful for us or for ubuntu. so yes it sounds good but we would appreciate if we waited a few days longer before merging this to sort things out.

have you seen my comment about the glesv2 issue?
also the bionic_file struct has a little issue which causes crashes in most applications (so far only seen on exit). it think it might be worth considering the inclusion of 7891cf2 directly here.

thanks for the hard work!

[ghosalmartin]

So are we going with this version of libhybris android 6 changes? What needs to be done to ensure legacy platforms are well supported?

[morphis]

@ghosalmartin Yes we will. This PR gives support for different linker versions where the right one is selected at runtime. We still support the 4.x based linker for legacy platforms and will use the 6.x one for newer ones.

[ghosalmartin]

Awesome, so whats stopping us? :P

[morphis]

Let me drop the debian/ part of this and then I will merge this in.

[bhush9]

Oh reminds me, I had to edit extract-headers.h to extract some more header files to get it built actually. I think it was set_abort_message.h and one more file..

[morphis]

@bhush9 Can you submit those changes as an additional PR?

[bhush9]

Sure, I can do that.

[morphis]

As CI has approved this now I intend to merge this later today if nobody else comments.

[stskeeps]

go for it