Skip to content
This repository has been archived by the owner on Jun 1, 2023. It is now read-only.

Let's port Xposed to N #230

Closed
abforce opened this issue Jul 3, 2017 · 96 comments
Closed

Let's port Xposed to N #230

abforce opened this issue Jul 3, 2017 · 96 comments

Comments

@abforce
Copy link

abforce commented Jul 3, 2017

@rovo89 We know that you're too busy to implement Xposed for N. No problem, just enjoy your life. But let us to do the job. You've got a high level of knowledge how stuff works on Art, and how Xposed should be implemented. but you've no time to do this.

On the other hand, there's lots of people been waiting for it, and of course they have much free time to do this great job, however they don't have a minimum and required insight on how just Art works and how it should be changed.

So, an idea would be sharing the knowledge by you, implementing it by us. Unfortunately Art doesn't come with a minimum and acceptable documentation, which drastically slows down the research on how stuff works. So, if you consider taking your time to write a brief and techninal writeup on the how, we'll have a good chance to keep the Xposed up to date with the latest release of Android.

What do you think? Do you have time for this or would prefer to still enjoy your life (and possibly playing with your puppies)?

Thanks in advance.

@rovo89
Copy link
Owner

rovo89 commented Jul 3, 2017

So, an idea would be sharing the knowledge by you, implementing it by us.

The actual implementation is the smallest part of developing Xposed. Getting the knowledge about how the ART compiler and runtime works and finding ways to integrate Xposed with maximum compatiblity is the most time-consuming and complex part. Books could be written about the small, yet very important details that have to be considered. Sometimes when I understood how a particular implementation works and which tricks they're using, I indeed think I should write something up about it, but that would take me even more time (which I currently don't have).

@roshanravan
Copy link

@Revo89 its very good to see that you are still pursuing beloved xposed, thank you for that.

Are you also looking into developing xposed for Android O too? Maybe Like Marshmallow some things even made it possible to integrate Xposed in a more elegant way in Android O and a workaround for JIT & AOT?! (We could only hope for)

We know your time is so little and there are many difficulties but seeing you replying replying gives xposed fan lots of hope

@abforce
Copy link
Author

abforce commented Jul 18, 2017

@rovo89 I just got my hands dirty. Please take a look at my copy: https://github.com/abforce/xposed_art_n. Everything works smoothly and flawlessly but XPrivacy. It breaks the boot process. I've posted an issue there along with its logcat output.

I'm heading toward building a built-in enabled Xposed firmware, so I was just porting commits relating to hooking methods not those relating to recompiling. See diff to get a clear insight on what I've done.

@roshanravan
Copy link

@abforce very interesting job.
If you are using pro version of xprivacy it has a stupid protection which checks for apps like LuckyPatcher and lvl emulation when device is booted. Maybe you break one of his checks during the boot. Can i ask about the firmware you said? It's for what device?

@abforce
Copy link
Author

abforce commented Jul 18, 2017

@roshanravan

Can i ask about the firmware you said? It's for what device?

Sure, please ask them on the issue page.

@roshanravan
Copy link

@abforce or may call you AliReza Aint you the person who started that project?
after reading your project this is my understanding that those modification should be done in N Rom? something like developing a custom Rom? right? in this way there is no need to xposed installer apk? if not how we check for module's update to control on which modules to be enabled/disabled?

i have OnePlust 3T and a Samsung Note4 n910c, unfortunately at the moment i use my OP3T for business purposes so i can not do testings on it , and there is no Android N for my Note4. but when your project gets stable enough and one made those ART modification to the N Rom i will flash it on my OP3T.

@rovo89
Copy link
Owner

rovo89 commented Jul 18, 2017

Congrats ;) Unfortunately, I can't review your changes, it's just too much and I don't remember all those details where I deviated from simply porting the Marshmallow changes and which additional changes were required to make it reliable (i.e. to ensure that the hooks are called even with compiler optimizations).

As you already mentioned, your changes could only work for ROMs compiled from scratch with Xposed already active, so people should understand that this won't work (at least not reliably) on their stock or otherwise pre-compiled ROMs.

@abforce
Copy link
Author

abforce commented Jul 18, 2017

@rovo89 Thanks for your prompt feedbacks. Surprisingly by porting M changes to N everything works well, EnableXposedHook, redirecting calls to proxy handler and then going back to Java part and calling callbacks. I believe that hooking functionality works possibly well. But there's a problem when GC runs, specifically I suspect that heap or stack will be malformed when XPrivacy is enabled.

As I traced down the error, the problem seems to be related to visiting roots when GC marks objects. Moving form M to N, just one of your commits I couldn't port directly, that of ArtMethod::VisitRoots.

Also enabling hook resources causes the problem when GC runs, are you sure your modified Object.Clone() doesn't cause memory issues?

I can't review your changes, it's just to

No, that's not too much. Please only review those parts relating to visiting roots. I'll heartily appreciate your effort. There's lots of people waiting for my custom ROM.

Just a moment please stop playing with your puppies and give them a rest. They really do need it.
Just joking, don't take it personally 😆

@dragonpt
Copy link

Congrats, this indeed seems just great
Kinda surprised, the M changes do work.

Either way i may give a look at building aosp with "patched" art

@roshanravan
Copy link

Thanks to @abforce and BlackSoulxxx
There is a flashable xposed for N available here:
https://forum.xda-developers.com/xposed/unofficial-xposed-v87-unofficial-yes-im-t3639192
It seems like it's for AOSP ROMs
Good job, we know Revo89 also working on different release from scratch but it will be so good if he helps and guide. As he is grand master.

@dragonpt
Copy link

haha yup :)

@newstart
Copy link

https://forum.xda-developers.com/xposed/xposed-android-nougat-sdk-25-arm64-t3639221

Sorry Revo, they beat you to the punch.

@wanam
Copy link

wanam commented Jul 19, 2017

Be polite people.

@Ninlives
Copy link

@newstart Don't they use rovo's codes? It can't exist without his work on Marshmallow, I don't think that's a punch.

@newstart
Copy link

Here are more builds, more stable.... surprisingly.

https://drive.google.com/drive/folders/0B5ePoKNP9UFtWGRVN2JlazhYU2c

@aviraxp
Copy link

aviraxp commented Jul 20, 2017

@newstart They use @abforce's source entirely. The "mod" they made is just about building.

@dragonpt
Copy link

Exactly

@rovo89
Copy link
Owner

rovo89 commented Jul 20, 2017

Surprisingly by porting M changes to N everything works well,

From my experience, I can tell you that it's indeed not that hard to get the basics done, but it takes a hell lot of time for the details. It's like the 80-20 rule, except that it's more like 95-5 or so.

The hooking approach I have designed for the previous releases is rather stable and can be applied for newer releases as well. I assume you needed most of the time to check all the places where IsProxyMethod() and IsDirect() is called and decide whether they should consider the new method modifier or not (i.e. use parameter true or not). If you didn't do that, you might have missed some places where they have newly introduced such calls. Same for methods like InstallStubsForMethod(), which needs to operate on the original original method even when they're called for a hooked method. You'd need to check for more places where this could be relevant. That's still in the range of days, but if you miss to do it, you might see obscure crashes or misbehavior that are very hard to debug.

Besides that, you took an approach where hooks would only work properly for ROMs which are entirely compiled from scratch. On stock ROMs and even most custom ROMs (that didn't include your port while building) as well as for any apps the device might have compiled before. So you'd have to start all over again with a wiped device running a special ROM. If not, then optimizations (e.g. inlining) might prevent a method from being called at all, and hence hooks wouldn't work. The worst thing about this is that the exact behavior would depend on the ROM, or even how the user uses the device (now that apps are compiled based on profiles). Modules would fail randomly and the module developers would be wondering what they did wrong (answer: nothing). If not all of the prerequisites are fulfilled, it's a pretty unreliable API. Imagine Google published an update where the OnClickListener of a button wasn't called sometimes, unless you compiled your ROM from scratch... For me personally, this is a big limitation.

There is a flashable xposed for N available here:
https://forum.xda-developers.com/xposed/unofficial-xposed-v87-unofficial-yes-im-t3639192
It seems like it's for AOSP ROMs

Their code is @abforce's port, with all the limitations I listed above. 🙄 It doesn't make sense at all to publish flashable ZIPs with this, as the port is only intended for recompiling a ROM from scratch.
And as I feared, there is not a single warning that this is an incomplete port that will often fail to fulfill the API contract. 😠 That's bad for developers and users, they should at least know it's incomplete so they can decide whether they want to support/use it. I would never dare to do something like that without a big fat warning. Actually, as I'm rather perfectionist, I wouldn't publish an incomplete version at all, reliablity is very important for me.

I just checked my Git log and I had the basic hooking (as in this port) ready in October. Since then, I have worked on removing the limitations. To make it more reliable on pre-compiled ROMs, I could have ported the recompiling part as well, however that's still not ideal. As explained somewhere above, we have JIT now, which allows us to keep all the optimizations, only invalidating (and possibly recompiling) the methods which are directly affected by hooking. This is quite challenging, as it's more than just copy & paste. And until this isn't done, I won't publish anything. 9 months for that (until now) is really a long time, I know, but hey, it's my spare time and e.g. due to moving to a new apartment, I couldn't work on Xposed for several months (!) at all, and had only very limited time otherwise.

As I traced down the error, the problem seems to be related to visiting roots when GC marks objects.

Also enabling hook resources causes the problem when GC runs, are you sure your modified Object.Clone() doesn't cause memory issues?

I just commented on your diff. There's no real difference in my code for these two methods, so you'll have to debug it yourself. I know this can be hard, as the real culprit can sometimes be in a totally different place. Even if it ends up to be a single character that needs to be changed, you can easily spend dozens of hours on debugging one issue. That said - if you're seriously interested in contributing, this might be a good challenge. 😉

@aviraxp
Copy link

aviraxp commented Jul 20, 2017

Totally agree with rovo, from my test, my module (which hooks many fundamental classes like android.view.ViewGroup) would fail randomly with many native crashes from the hooked app. Although we can install it (even flash it) with a not pre-odexed rom and cleaning every partition to clear odex files, it is not stable at all and cannot be used for production environment.

@dragonpt
Copy link

Rovo of course Im totally with you on this.

Also prebuilt stuff its not the way, I'll be doing (even with all the limitations ) patch our ROM source, that's really the only way to at least debug and not just flash an miracle prebuilt stuff.zip

We all appreciate your work rovo, and I know how long this stuff can takes when done on our limited free Tim
Cheers

@rovo89
Copy link
Owner

rovo89 commented Jul 20, 2017

And just to add to my reply above: Imagine I published my current WIP now. I bet that some people would publish it as their build, without any warnings that it's incomplete and not intended to be used productively.

@dragonpt
Copy link

Surely !

@randomhydrosol
Copy link

randomhydrosol commented Jul 20, 2017

Can confirm this. I build with full dexpreopt and dexpreopt_pic and the unofficial xposed causes multiple soft-reboots repeatedly even when modules are not enabled on my ROM

@dragonpt
Copy link

dragonpt commented Jul 20, 2017

Just saw xda's "Purify" and there's an higher BS there..
For real...
Asking donations in your behalf using ABForce work patch up & ofc the whole rovo's work

@Spektykles
Copy link

And there is an XDA News article backed it up...

@dragonpt
Copy link

Yup top notch ...
/irony

@mo3rfan
Copy link

mo3rfan commented Jul 20, 2017

Slightly offtopic, but I was also interested in hacking around to get Xposed to work on Nougat. The only thing that put me off is the huge resource requirements for the dev environment. 100 GB+ of disk space/bandwidth and large RAM requirements just to get AOSP to compile! Is there a better way to work with android/ART? How do you guys do it? @rovo89 @abforce

@dragonpt
Copy link

Its the normal environment to get aosp compiled .. You could get away with just 8gb Ram a fairly CPU and ofc the HDD space (ssd even better) if you don't have those requirements, maybe search for a cloud/build server solution

@rovo89
Copy link
Owner

rovo89 commented Jul 20, 2017

Yep, that quite unfortunate, but that's because AOSP source is so huge. The only hint I can share is to initialize like this:

repo init -u https://android.googlesource.com/platform/manifest -g default,-device,-darwin,-cts,platform-linux -b <branch>

With the -g parameter, you can limit the repos that should be downloaded to certain groups. Excluding some groups can avoid to download stuff that's not needed for ART anyway (but much other stuff is actually required).

Besides that, if you're planning to check out multiple versions of AOSP (e.g. Lollipop, Marshmallow and Nougat), it helps to symlink the ".repo/project-objects" folder to a shared place before fetching the source code.

@Vynikal
Copy link

Vynikal commented Jul 20, 2017

So... if the Xposed code is compiled with the ROM, is it stable without any said limitations?
If yes, it would be nice to squash the changes to one commit to be applied to source tree, so ROM developers could use it. Just a suggestion.
EDIT: As I see, it would mean change the ART repository according to abforce's version and replacing app_process package, am I right?

@amakuramio
Copy link

amakuramio commented Jul 29, 2017

I did the following to fix it>

  1. Uninstall Twitter
  2. Flash Xposed SDK 24 26.07 (purifyOS)
  3. Reboot
  4. Install DVDandroid Xposed 27.07
  5. Reboot
  6. Install Xprivacy, enable it
  7. Reboot
  8. It works, but some app force close
  9. Reboot to recovery, wipe dalvik/art cache

fixed, no problems, Superkernel Android 7.0 stock, S7 Edge

@abforce
Copy link
Author

abforce commented Jul 29, 2017

People! Please keep calm and wait and don't try to leave chatty comments. Please do leave your knowledge not pointless comments. My ART copy works well (actually thank to @rovo89's work for Android M, I've done nothing), but there's random crashes when GC runs. I think there's a problem in memory allocation on the heap. I'm sure enough to say that hooking functionality works pretty good. So we should move our attentions to where memories are allocated. After reviewing stacktraces all crashes somehow is related to CMS (Concurrent Mark and Sweep) GC algorithm. I'm not saying that the CMS itself needs modification, no no. I just think that methods called VisitRoots should be more focused on.

For example, the following methods could be the main culprit because of which we're receiving SIGABRT and SIGSEGV. Please inspect them elaborately.

  • ArtMethod::VisitRoots in runtime/art_method-inl.h
  • VisitQuickFrame() in runtime/thread.cc

If anyone has a deep insight on how GC works in ART, his/her input will be welcomed. If no one has, just get your hands dirty, dig down until you reach the first commit of ART. :)

@wanam
Copy link

wanam commented Jul 29, 2017

@abforce the visitRoots issues seems to be fixed with @ahronshor commit: abforce/xposed_art_n@94ba5a5

But I'm still getting a few cases of mark/sweep crashes.

@wanam
Copy link

wanam commented Jul 30, 2017

@abforce I also fixed resources hooking for Android 7.1 on OOS, based on the same solution of @ahronshor ahronshor/XposedBridge@0aec3f9#diff-a34e2ca228dbe63004fcc32e3bf8581a , you need to look for the correct signature of the method "getOrCreateResources", at least OOS does not have any signature of the method that have the resourcekey as a second parameter, you need to do something like this:
for(Object ob:param.args){ if (ob != null && ob instanceof ResourcesKey){ key=ob; } }
here ahronshor/XposedBridge@0aec3f9#diff-c91560976a56d92e8c015d1bdd70ee46R260

@abforce
Copy link
Author

abforce commented Jul 31, 2017

@wanam

the visitRoots issues seems to be fixed with @ahronshor commit:

I can't believe it fixed the problem. I'm more thinking that it just suppressed the problem preventing us from booting the device, not fixing the problem in an ideal way. Since we've got little or no knowledge of ART internals, our works little or more can be thought of as trials and errors. That being said, fixing a problem may cause another issue elsewhere.

But I'm still getting a few cases of mark/sweep crashes.

What do you think about them?

@wanam
Copy link

wanam commented Jul 31, 2017

@abforce

I can't believe it fixed the problem. I'm more thinking that it just suppressed the problem preventing us from booting the device, not fixing the problem in an ideal way. Since we've got little or no knowledge of ART internals, our works little or more can be thought of as trials and errors. That being said, fixing a problem may cause another issue elsewhere.

I agree, i'm not sure if it's an ideal solution, but at least it didn't cause any other visible issue yet.

What do you think about them?

I got that crash twice last week, i don't know how to reproduce it again, it crashed here https://github.com/abforce/xposed_art_n/blob/master/runtime/gc/collector/mark_sweep.cc#L415 but i have no idea why, the stack trace does not tell much about when it started.

@C3C0
Copy link

C3C0 commented Aug 8, 2017

@ahronshor @wanam
Regarding modifications to the XposedBridge new resources handling. I tried your version of XposedBridge which is basically distributed with PurifyOS xposed packages but I found out that resources injection using fake res ID and referencing them in a module doesn't work in it. I did not study the patches in detail as I came up with somewhat a different solution a couple of weeks ago (was for my personal use but now I thought it was a time to share it).
C3C0/XposedBridge@605fdfe
Resource injection works with this version on AOSP SDK24/25. Not sure if anything else is needed - I did not investigate any further as those changes were enough for my personal scenario.

@avently
Copy link

avently commented Aug 8, 2017

@C3C0 will you make a pull request?

@C3C0
Copy link

C3C0 commented Aug 10, 2017

@avently
It doesn't make sense as it would conflict with their implementation.

@abforce
Copy link
Author

abforce commented Aug 16, 2017

@rovo89 What's up man? Is it WIP or zzZ? Is it needed to get my hands dirty for the second time? 😆

@rovo89
Copy link
Owner

rovo89 commented Aug 17, 2017

WIP:
grafik

@dragonpt
Copy link

;-)

@Jedarc
Copy link

Jedarc commented Aug 17, 2017

@rovo89

var answer = prompt('Has anyone told you today that loves you?');

if (answer === 'no') {
   for(var i = 0; i < 1000; i++) {
      console.log('I love you!');
   }
}

@randomhydrosol
Copy link

I love you

@randomhydrosol
Copy link

randomhydrosol commented Aug 17, 2017

Now, answer = yes

@randomhydrosol
Copy link

Hey rovo
You're a great dev and we know that
You can work on xposed as you like and when you like
You don't need to constantly show us proof
Some impatient guys do exist but please don't force yourself into developing

@ghost
Copy link

ghost commented Aug 18, 2017

Yes, there are thousands of superusers waiting patiently for your release of Xposed for Android N
Don't push yourself too hard. We love you sir, keep up the good work

@dragonpt
Copy link

Rovo just close this...

Just spammers here.

People no ETA, when its done its done!
Jesss

@MarcinOrlowski
Copy link

MarcinOrlowski commented Aug 21, 2017

it's my spare time

@rovo89 What prevent you from starting community funding for Xposed? This project got a real value for many people and that value can be expressed in money. I'd love to donate to the project to ensure you can focus on xposed for N, instead of doing anything else to earn money to pay your bills.

@randomhydrosol
Copy link

randomhydrosol commented Aug 21, 2017 via email

@MarcinOrlowski
Copy link

MarcinOrlowski commented Aug 21, 2017

The same thing he would do after completing any other project. Focus on something else.

@amakuramio
Copy link

amakuramio commented Aug 21, 2017

so much spam from people who don't know any shit.
if you have nothing to contribute to this topic like development or bug reports then fuck off.
and the poor people who try to say to make it a fund project, yet wont even donate 0.01$. like srsly?

@dragonpt
Copy link

Yup...

@MarcinOrlowski
Copy link

MarcinOrlowski commented Aug 21, 2017

"poor people"

@amakuramio There's no @rovo89's crowdfunded project alive AFAIR to donate to, so stop playing smartass rich dude while you in fact got not much to say. It's none of your business what people do with own money. FYI: I personally agree to rovo89's statement about the current outbreak of N's "compatible" builds, so I will not sponsor these forks. But I would rovo89 given the way to do that.

@dragonpt
Copy link

?? Why are you quoting me on this?

That didn't come from me lol

Either way @rovo89 again just close this...

Too many spammers here

@newstart
Copy link

@abforce Let's port Xposed to O 😂.

O Will be released with the eclipse lol.

https://www.android.com/eclipse/

@rovo89 rovo89 closed this as completed Aug 21, 2017
@rovo89
Copy link
Owner

rovo89 commented Aug 21, 2017

This thread really doesn't bring any progress, so I'll close it. And no, I wouldn't give up my job just to spend more time on Xposed, it's still just a hobby. 😀

@newstart
Copy link

newstart commented Aug 21, 2017

No worries Rovo, abforce has given us what we want. Take all the time in the world.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests