ReactiveCocoa does not work well with NewRelic agent (or vice versa)

[nikita-leonov]

I didn't investigate it deeply yet, but due to invocation swizzling in RAC and some internal behaviors in NewRelic (http://newrelic.com) agent cause application to froze right after launch. Apparently it is going into infinity loop:

• CPU consumption — https://www.evernote.com/shard/s70/sh/2397ba6c-6aad-447a-bf2b-0d2bee044522/941ccb0eb4d15339ce39f9c790cc72fb
• Stack for Thread 1 — https://www.evernote.com/shard/s70/sh/d7e4c61a-90d6-43ed-a91e-f0bde38bcffe/2deb01702f34b219409d3cfe5dc3ae58

I reported it to NewRelic as well with issue id 117914

[jspahrsummers]

Without seeing the code, it's hard to know where the issue specifically lies. However, an easy workaround would be to avoid the use of -rac_signalForSelector: on affected classes.

[nikita-leonov]

I think that not using rac_signalForSelector will affect our architecture in a negative way. I would rather fix this issue. As NewRelic is closed-source I am not able to fix it myself, but I already started work with NewRelic team and provided them necessary prerequisites to reproduce and fix. Working on a simplified application to reproduce an issue as well.

[nikita-leonov]

I was able to reproduce issue and delivered example application to Newrelic team. There is no answer yet why it is happening, and it is hard to investigate on what side things goes wrong, as loop happening after RAC swizzling in Newrelic code. The code to reproduce is as simple as:

    func application(application: UIApplication, didFinishLaunchingWithOptions launchOptions: [NSObject: AnyObject]?) -> Bool {        
        NewRelic.startWithApplicationToken("put your newrelice token here")
        window?.rootViewController?.rac_signalForSelector("viewDidAppear:")
        return true
    }

Newrelic team aware about this issue, so if there will be some issues related to RAC I think they will follow up here.

[PrideChung]

It looks like RAC and Newrelic swizzling a same method and somehow causing an infinite loop. I encountered a similar problem while using RAC and BlocksKit together before and it's hard to solve. People claim method swizzling is evil for a reason.

[powerje]

I'm not sure if this is helpful or not but I think Aspects had a similar issue with New Relic:
steipete/Aspects#21

It looks like @steipete's fix will be to re-implement how Aspects performs method swizzling based on this New Relic blog post: http://blog.newrelic.com/2014/04/16/right-way-to-swizzle/

[steipete]

Yes, they proposed a different way of swizzling which has it's own drawbacks... haven't actually gotten into updating Aspects for that, especially also because NewRelic's framework is closed source, obfuscated and also protected against reverse engineering - so debugging with it is next to impossible. Their support also ignored my issues for the longest time.

[nikita-leonov]

According to NewRelic team swizzling in RAC done incorrectly, that causing NewRelic to fail. I requested them to post here more details about NR internal swizzling and how they see RAC swizzling to be fixed. Without that information I do not believe this task will move forward.

[nikita-leonov]

Actually link provided in support ticket by NewRelic are the same as @powerje posted.

[nikita-leonov]

Application that reproduce an issue available here— https://github.com/BuddyHOPP/NewrelicInfiniteLoopIssue

[buchananbryce]

The issue lies in how New Relic dynamically swizzles system methods. It depends on the method's _cmd parameter to match the system's default signature. The best way to solve this problem is to ensure the cmd value of swizzled methods isn't populated with "@selector(rac_alias<method_name>)" and instead populated with "@selector(<method_name>)". Not sure if this is feasible with the way reactive cocoa does swizzling.

[jspahrsummers]

There's a really long-winded explanation here, but the TLDR is that it's really not possible for us to affect _cmd in the necessary way. -signalForSelector: requires a lot of tricks to work, especially in the presence of KVO.

@nikita-leonov As I mentioned above, it's probably in your best interest to avoid it. But you only need to avoid it for classes affected by NewRelic.

/cc @kastiglione @Coneko @joshvera @keithduncan who know a lot about the swizzling too

[nikita-leonov]

@jspahrsummers unfortunately it is almost like say "do not use signalForSelector at all" It will also cause us to refactor a lot of things that are already in place.

It will be interesting to know actually details why RAC is incompatible with approach that NewRelic suggested, as it looks pretty legit.

[kastiglione]

unfortunately it is almost like say "do not use signalForSelector at all"

@nikita-leonov Does NewRelic document what they swizzle?

[nikita-leonov]

@kastiglione it would be nice question to ask, I will do so. In my understanding they swizzle almost everything, as they measure time of execution of calls, but I actually do not know how deep they are going down to. I will post here more information, as it may be helpful for temporary workaround.

[buchananbryce]

Here's a table of the things we swizzle. We will also swizzle anything that subclasses these methods. https://docs.newrelic.com/docs/mobile-monitoring/mobile-sdk-api/new-relic-mobile-sdk-api/working-ios-sdk-api#instrumenting

[nikita-leonov]

@buchananbryce Thanks!

[kastiglione]

The best way to solve this problem is to ensure the _cmd value of swizzled methods isn't populated with "@selector(rac_alias_<method_name>)" and instead populated with "@selector(< method_name>)".

@buchananbryce Why not just add a check that strips "rac_alias_" prefixes? Not elegant, but I'm sure it would make happy customers. Or if that's too specific, maybe provide an API for apps to map swizzled selectors. You say yourself "The issue lies in how New Relic dynamically swizzles system methods.". I'm making lots of assumptions, but that's because I'm not much clear of "why".

[buchananbryce]

It's not really feasible to add a string compare for every possible framework that could be swizzling, and again, it's not really feasible to expect any person to know which prefixes to search for... I feel like 'there be dragons' going down this rabbit hole for a solution.

The 'why' is New Relic relies on the _cmd parameter to match it's default value, to allow us to instrument a variable number of methods without any sort of user intervention to make it happen, and ideally, when swizzling, the _cmd value should always match the name of the method that's being executed.

I see that you are swizzling the forwardInvocation method; would it be possible for you to manipulate the _cmd parameter in the invocation's parameter list before forwarding it on at that point?

[jspahrsummers]

I see that you are swizzling the forwardInvocation method; would it be possible for you to manipulate the _cmd parameter in the invocation's parameter list before forwarding it on at that point?

@buchananbryce Hmm, that's a good question that hadn't occurred to me before. It'd certainly be worth a shot!

[jspahrsummers]

Ah, I remember now why that won't work. The issue is that, when we have a rac_alias_ selector, we use NSInvocation to invoke it with the correct conventions. That doesn't give us a hook to change _cmd before the method sees it.

The need to use NSInvocation is, in fact, the reason that we even have rac_alias_ selectors in the first place (vs. saving IMPs). We support intercepting methods of arbitrary signatures, so it's impossible to handle all the possible cases ourselves.

TLDR: I don't think there's an easy workaround here. Where it's an issue, I'm afraid that avoiding -rac_signalForSelector: is the only option. It's really not that bad to go without it—most uses can be replaced with a simple RACSubject property.

[kastiglione]

The code set the invocation's selector to the RAC alias. See NSObject+RACSelectorSignal.m#L49. EDIT: Yeah, what @jspahrsummers said.

[kastiglione]

It would be great if NSInvocation made -invokeUsingIMP: public, that might solve our problems.

[kastiglione]

We support intercepting methods of arbitrary signatures, so it's impossible to handle all the possible cases ourselves.

The thought I had on this is we could special case a handful of very common signatures. Somewhat like the block trampoline, except maybe with the addition of handling some non-object types. If the selector is irregular, we resort to _objc_msgForward.

From New Relic's docs, the signatures on UIViewController are twofold: void -> void and BOOL -> void.

[jspahrsummers]

@kastiglione I'm not a big fan of that inconsistency. Although the current behavior is pretty shitty with regard to _cmd, it's at least consistent across all observed selectors.

[kastiglione]

It's not really feasible to add a string compare for every possible framework that could be swizzling

@buchananbryce I feel like this may be a fallacy. If there are that many frameworks that are causing problems that you can't special case them, then seems to me you have a larger problem on your hands. If there aren't that many, then why is it not feasible?

Btw, Apple and WebKit both special case by bundle ID to work around problems, but of course that's not an apples to apples comparison.

[kastiglione]

@jspahrsummers I'm just thinking of helping out RAC developers. It sucks, but at least subjects can be used as a workaround.

[grp]

You could fix this by replacing -[NSInvocation invoke] with libffi, no? Convert the type signature and ffi_call() the IMP you want.

[kastiglione]

I presume something like that would work. Libffi has been mentioned a couple of times, but no serious discussion (that I've seen).

[jspahrsummers]

libffi would probably work, yes, but it's also a large, scary, and fragile dependency (especially on iOS) to bring in. I generally prefer to rely upon built-in Cocoa tooling where possible.

[bobspryn]

I assume there's been no changes on this on either side? Just ran into the same issue. We'll be reporting it to new relic, but it sounds like they aren't interested in changing it.

[jspahrsummers]

The RAC problem is extremely hard (if not impossible) to fix without libffi. The suggested workaround is to use RACSubject instead of -rac_signalForSelector: for affected code.

[bobspryn]

Yeah that's what we're doing. Just cumbersome. Love signalForSelector for lifecycle methods. Understood on not wanting to introduce a dependency. Maybe @buchananbryce will throw us some love and add an exception for RAC.

[fightingmonk]

Hi @sprynmr. We're not opposed to adding special case support for RAC. We do have to consider the performance implications and how popular the framework is. Please do file a report / request so we've got your vote counted.

[bobspryn]

@fightingmonk Noted. We sent our request along through proper channels. Thanks!

[mdarnall]

I'm seeing this issue when using rac_signalForSelector: to manage the subscription only until a view disappears. I've filed a support case with New Relic asking for them to special case RAC.

RACSignal *willDisappear = [self rac_signalForSelector:@selector(viewWillDisappear:)];
[[mySignal takeUntil:willDisappear] subscribeNext:^(id x){}]]