Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Support Kotlin/Native in MockK #58

Open
olonho opened this issue Apr 8, 2018 · 51 comments
Open

Feature: Support Kotlin/Native in MockK #58

olonho opened this issue Apr 8, 2018 · 51 comments

Comments

@olonho
Copy link

olonho commented Apr 8, 2018

For multiplatform projects it would be really nice if MockK would support Kotlin/Native as well. If some missing functionality on K/N side is needed, please let us know, and we will add it.

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 8, 2018

I wonder how that can be done technically.

Does KN support any kind of instrumentation?

We need at least somehow inject a check of boolean in each method that defines if the function should be intercepted or something alike. This would be compile-time instrumentation/compiler plugin, of course, the runtime is even better. I've heard Kotlin way is to write compiler plugins

@olonho how is it done?

@olonho
Copy link
Author

olonho commented Apr 8, 2018

Runtime instrumentation is indeed likely out of the question. Unless we will do something like https://en.wikipedia.org/wiki/DTrace with lightweight probes.

Compile-time wise, there are options. Most obvious one is a source transformation, but it isn't really nice. Another one is to implement compiler plugins API, and do something like https://github.com/JetBrains/kotlin-native/blob/master/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/lower/TestProcessor.kt on IR.

I think plugin compiler API is most sensible option, we can try to implement an extension point which runs an additional pass, if certain plugin is available. Do you find acceptable option to implement simple IR transformer?

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 8, 2018

Yes, IR transformer seems to be the way to go.

One more possible solution is to dynamically intercept functions after code generation i.e. embedding some assembly code on the fly by using additional information that compiler provides and dispatching only needed functions. But this seems to be hard. (I believe it is similar to DTrace, just another level)

Many details and design decision to be discussed if IR transformer is selected.

I'll sketch just a few:

  • mocking can be done to any class in runtime. should we instrument all functions? what can be done better?
  • need instrumentation that can call original functions and return result instead of it. how to do that?
  • how to achieve minimum performance impact?
  • should compiler build transformed version only for testing? is there such possibility?
  • should be supported other mocking frameworks? sooner or later such interference may appear.

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 8, 2018

In general, I can admit that most of the functions of mockk itself are transferred to common modules. Besides implementing the interface for function interception there are no hard tasks to perform for porting it to any platform.

Additionally, I struggle with following problems:

  • JS support is experimental because it works quite badly with default parameters which are not supported by JS Proxy class. It would be nice to have solution for multiplatform mocking at the end.
  • some things users would like to have extra, alike mocking inline functions
  • Ideally I would like to fully support Kotlin for all platforms(Java, JS, Native, Android device testing) and possible features without any Java agents or Proxies, just via the processor, and only for legacy use additional Java agents and Proxies.

@olonho
Copy link
Author

olonho commented Apr 9, 2018

mocking can be done to any class in runtime. should we instrument all functions? what can be done better?
Probably, for Native we could compile with a special option, and insert call to some uniform listener with callee information. Then, actual behavior can be overiden in the runtime (slow, but will work).
Another option is limit mockability scope to only those classes seen by the compiler.

need instrumentation that can call original functions and return result instead of it. how to do that?

This shall be straightforward (as in test runner above) - you can create arbitrary complex IR instead of the original function.

how to achieve minimum performance impact?

Without dtrace-like mechanisms, I presume it will slow down execution anyway, but smartness may come from the way of calling mocked function (smth like dynamic table of invocation targets, where either original implementations lie, or we could replace it with custom pointer when mocking).

With that design, when compiled with the dynamic targets table, we could create following additional structures:

  • map of function name to an index in tables below (i.e. list of (const char*, index) pairs) - name table (NT)
  • original function pointers table (OFP)
  • actual function pointers table (AFP)

On call site of every function we call indirectly via the actual function table. Mockers can just modify this table, using elements of the original table to figure out where to call if need original implementation. This would kill performance (every call is indirect, and no inlining), but could allow rather flexible mocking without too deep integration into compiler. WDYT?

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 9, 2018

Performance is a difficult thing to reason as many things might be optimized on CPU level e.t.c. Without performance testing and optimization I don't think we can prove anything here in discussion.

Dynamic tables is kind of classics but require I believe more compiler/runtime rewrite.

Better do just interception via similar approach i.e. have a table that is checked at the beginning of every function and if it is filled-in then special call to mocking framework is performed:

  • call sites don't need to know anything about it
  • can be implemented at a higher level IR (in case you have such). best IMHO if such approch would be portable between platforms.
  • as dynamic call can be two level: class lookup and then function. good performance because do just one check at the begning and table of classes can be cached, this assumes most classes are not mocked.
  • can support mocking inline functions
  • it should translate to few additional asm instructions with guaranteed branch prediction if done carefully. (hope so)

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 10, 2018

Although one thing that is better with dynamic dispatch is that it allows calling original methods, which is required by spies. See no way now to do it with interceptors without overcomplication of it.

oleksiyp added a commit that referenced this issue Apr 10, 2018
@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 10, 2018

@olonho I have following error here:

* Where:
Build file '/home/oleksiyp/workspace/mockk/dsl/native/build.gradle' line: 24

* What went wrong:
A problem occurred evaluating project ':mockk-dsl-native'.
> tried to access method org.jetbrains.kotlin.gradle.plugin.KotlinPlatformImplementationPluginBase$Companion.whenEvaluated(Lorg/gradle/api/Project;Lkotlin/jvm/functions/Function1;)V from class org.jetbrains.kotlin.gradle.plugin.KotlinNativePlatformPlugin

idea.log

@olonho
Copy link
Author

olonho commented Apr 11, 2018

We're looking on this issue.

@oleksiyp
Copy link
Collaborator

@olonho I use following kotlin plugin:

buildscript {
    ext.kotlin_version = '1.2.21'
    repositories {
        mavenCentral()
        jcenter()
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
    }
}

@vvlevchenko
Copy link

vvlevchenko commented Apr 11, 2018

workaround for you issue could be following:

  • switch to kotlin-plugin 1.2.20 (because kotlin-native plugin depends on this version)
  • add kotlin-native plugin dependency to main build.gradle

@oleksiyp
Copy link
Collaborator

Thanks. Will try this evening

oleksiyp added a commit that referenced this issue Apr 11, 2018
@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 11, 2018

As in my project DSL is extracted to specific module I need to port it first. Added dsl/CMakeLists.txt with common part + InternalPlatform borrowed from JS. Now it fails with following error:

exception: java.lang.AssertionError: Built-in class kotlin.Function23 is not found
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns.getBuiltInClassByName(KotlinBuiltIns.java:431)
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns.access$700(KotlinBuiltIns.java:57)
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns$4.invoke(KotlinBuiltIns.java:141)
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns$4.invoke(KotlinBuiltIns.java:138)
	at org.jetbrains.kotlin.storage.LockBasedStorageManager$MapBasedMemoizedFunction.invoke(LockBasedStorageManager.java:408)
	at org.jetbrains.kotlin.storage.LockBasedStorageManager$MapBasedMemoizedFunctionToNotNull.invoke(LockBasedStorageManager.java:483)
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns.getBuiltInClassByName(KotlinBuiltIns.java:424)
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns.getBuiltInClassByName(KotlinBuiltIns.java:466)
	at org.jetbrains.kotlin.builtins.KotlinBuiltIns.getFunction(KotlinBuiltIns.java:561)
	at org.jetbrains.kotlin.builtins.FunctionTypesKt.createFunctionType(functionTypes.kt:199)
	at org.jetbrains.kotlin.builtins.FunctionTypesKt.createFunctionType$default(functionTypes.kt:194)
	at org.jetbrains.kotlin.builtins.SuspendFunctionTypesKt.transformSuspendFunctionToRuntimeFunctionType(suspendFunctionTypes.kt:57)
	at org.jetbrains.kotlin.serialization.KonanDescriptorSerializer.type(KonanDescriptorSerializer.kt:501)
	at org.jetbrains.kotlin.serialization.KonanDescriptorSerializer.typeId(KonanDescriptorSerializer.kt:475)
	at org.jetbrains.kotlin.serialization.KonanDescriptorSerializer.typeParameter(KonanDescriptorSerializer.kt:465)
	at org.jetbrains.kotlin.serialization.KonanDescriptorSerializer.functionProto(KonanDescriptorSerializer.kt:281)
	at org.jetbrains.kotlin.serialization.KonanDescriptorSerializer.classProto(KonanDescriptorSerializer.kt:111)
	at org.jetbrains.kotlin.backend.konan.serialization.KonanSerializationUtil.serializeClass(KonanSerializationUtil.kt:173)
	at org.jetbrains.kotlin.backend.konan.serialization.KonanSerializationUtil.serializeClasses(KonanSerializationUtil.kt:193)
	at org.jetbrains.kotlin.backend.konan.serialization.KonanSerializationUtil.serializePackage(KonanSerializationUtil.kt:219)
	at org.jetbrains.kotlin.backend.konan.serialization.KonanSerializationUtil.serializeModule$backend_native_compiler(KonanSerializationUtil.kt:262)
	at org.jetbrains.kotlin.backend.konan.KonanDriverKt$runTopLevelPhases$3.invoke(KonanDriver.kt:88)
	at org.jetbrains.kotlin.backend.konan.KonanDriverKt$runTopLevelPhases$3.invoke(KonanDriver.kt)
	at org.jetbrains.kotlin.backend.konan.PhaseManager$phase$$inlined$with$lambda$1.invoke(KonanPhases.kt:140)
	at org.jetbrains.kotlin.backend.konan.PhaseManager$phase$$inlined$with$lambda$1.invoke(KonanPhases.kt:119)
	at org.jetbrains.kotlin.backend.konan.util.UtilKt.profileIf(util.kt:34)
	at org.jetbrains.kotlin.backend.konan.PhaseManager.phase$backend_native_compiler(KonanPhases.kt:139)
	at org.jetbrains.kotlin.backend.konan.KonanDriverKt.runTopLevelPhases(KonanDriver.kt:84)
	at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:61)
	at org.jetbrains.kotlin.cli.bc.K2Native.doExecute(K2Native.kt:39)
	at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:108)
	at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.java:52)
	at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:92)
	at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:70)
	at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:36)
	at org.jetbrains.kotlin.cli.common.CLITool$Companion.doMainNoExit(CLITool.kt:157)
	at org.jetbrains.kotlin.cli.common.CLITool$Companion.doMain(CLITool.kt:148)
	at org.jetbrains.kotlin.cli.bc.K2Native$Companion$main$1.invoke(K2Native.kt:183)
	at org.jetbrains.kotlin.cli.bc.K2Native$Companion$main$1.invoke(K2Native.kt:174)
	at org.jetbrains.kotlin.backend.konan.util.UtilKt.profileIf(util.kt:34)
	at org.jetbrains.kotlin.backend.konan.util.UtilKt.profile(util.kt:29)
	at org.jetbrains.kotlin.cli.bc.K2Native$Companion.main(K2Native.kt:176)
	at org.jetbrains.kotlin.cli.bc.K2NativeKt.main(K2Native.kt:188)
	at org.jetbrains.kotlin.cli.utilities.MainKt.main(main.kt:27)

I have functions of 22 args + continatuion arg in API.kt:

inline fun <reified T : suspend (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, A22) -> R, R, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, A22> CapturingSlot<T>.coInvoke(

Is it supported?

oleksiyp added a commit that referenced this issue Apr 11, 2018
@oleksiyp
Copy link
Collaborator

Wow! After fixing this error DSL compiled in CLion and there is a possibility to run it till the moment actual implementation is needed. Optimistically tomorrow evening will port main implementation. After that only method interception is left.

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 12, 2018

@olonho @vvlevchenko what to do with following message?

error: the feature "multi platform projects" is experimental and should be enabled explicitly

Is it possible to turn it on in CMakeLists.txt?

@olonho
Copy link
Author

olonho commented Apr 12, 2018

To enable MPP please use -Xmulti-platform compiler option. But for build we'd recommend to use Gradle, not CMake.

@oleksiyp
Copy link
Collaborator

Then how I'll get IDE support if I use gradle?

@olonho
Copy link
Author

olonho commented Apr 12, 2018

Currently (for short period of time, we hope) it is somewhat problematic if you want IDE for Native-specific components, for common parts IDEA will work, and could be used in non-intelligent mode to edit Native-specific files as well. Longer term we will go Gralde way, so I just suggest to take that into account.

@oleksiyp
Copy link
Collaborator

And what about "suspend" 22 arg lambda? Should I raise ticket somewhere?

@oleksiyp
Copy link
Collaborator

oleksiyp commented Apr 12, 2018

I ported boilerplate implementation code. output

Following things are marked as TODO() and required to be provided by platform/implementation:

  • proxy for function interception (this will be replaced by dynamic dispatch)
  • instantiation empty object of an arbitrary class
  • random
  • identity hash code
  • copying all fields of from one object to another (used in spies)

As well to mock private properties / fun I use reflective calls. For coroutines I need runBlocking, don't know if it is supported.

Additionally, my CLion is not indexing any content except I already visited, even simple serach is not working.

@oleksiyp
Copy link
Collaborator

Besides that I added println('hi') statement to SuspendFunctionsLowering and compiled mockk using new compiler. It is working!

I can say that without knowledge of IR it is quite difficult to understand what each lowering does. I tried to figure out for Autoboxing because it is small and seems even that is not so easy

@olonho
Copy link
Author

olonho commented Apr 13, 2018

Regarding 23-parameters functions - guess this is will be fixed generally with https://youtrack.jetbrains.com/issue/KT-13764 in all Kotlin flavours.

Regarding IR transformation, likely what you may need is somewhat a mix of https://github.com/JetBrains/kotlin-native/blob/master/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/lower/BridgesBuilding.kt
and
https://github.com/JetBrains/kotlin-native/blob/master/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/lower/TestProcessor.kt
Autoboxing is not exactly what you need, as it modifies callsites, while bridge builder processes callees.

@stale
Copy link

stale bot commented Jul 23, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you are sure that this issue is important and should not be marked as stale just put an important tag.

@stale stale bot added the stale label Jul 23, 2019
@JanStoltman
Copy link

@oleksiyp Could you perhaps add important tag to this issue, as it definitely is an important one.

@stale stale bot removed the stale label Jul 24, 2019
@oleksiyp oleksiyp changed the title Native: Support Kotlin/Native in MockK Feature: Support Kotlin/Native in MockK Nov 1, 2019
@ShikaSD
Copy link

ShikaSD commented Dec 20, 2019

@oleksiyp
Hi, I was looking around for mocking for KMPP and found this issue.
Currently Kotlin seems to move towards uniform IR for all the platforms in compiler, so maybe it would be possible to actually try to implement a compiler plugin to allow mocking on every supported platform?

I was going through the issue and noticed two unfinished PRs (this and this).
They are great chunk of work, but it would be great if you could provide some line of thought what is required to be implemented to make mockk work without reflection?

@oleksiyp
Copy link
Collaborator

oleksiyp commented Dec 20, 2019

@ShikaSD basically interception of any call is needed. looks unrealistic on IR level.

I would add NOPs at start of every function and remembered all these locations. Then when mock is created, for each function that is related to the mock change NOP to JMP instruction. It might work 😃

@olonho what do you think?

@olonho
Copy link
Author

olonho commented Dec 21, 2019

Hmm, why do you believe it is not realistic to intercept calls on IR level? Simplest approach would be to remember all mocked functions and just modify their body IR by adding some MockK function calls in prologue and epilogue. Then you don't need to track all calls, and just focus on callee side.

@oleksiyp
Copy link
Collaborator

All mocked functions = all program, coz everything is covered with unit tests and everything is running in one run usually.

@olonho
Copy link
Author

olonho commented Dec 21, 2019

It's possible to see and modify IR of the whole program, depending on place of plugin in compiler pipeline (after deserialization) and if compiler caches are used.

@oleksiyp
Copy link
Collaborator

oleksiyp commented Dec 21, 2019

I tried in these PRs and even had some success to intercept one function or alike. Rather complex, but you are right, this might be achievable. Not sure about the performance of all this thing, how to optimize it in IR the way that overhead is minimal.

I was actually waiting for the promised refactoring of a compiler since the time of first PRs (tooling was not super pleasant and codebase quite complex, and I had not so powerful laptop 😃). Believe that is what is happening now, clean rewrite.

I would try again, but the problem is that I became involved very tightly into Cloud topic at work and spend all my free(to learn) and work time for this purpose.

@ShikaSD
Copy link

ShikaSD commented Dec 21, 2019

Correct me if I am wrong, but from my perspective, for mocking we can generate anonymous implementations/wrappers of interfaces/classes user is mocking.
Therefore interception is "just" implementation of these classes, delegating to mockk. I am probably missing something, but why does mockk actually need to intercept all the calls directly?

@ShikaSD
Copy link

ShikaSD commented Dec 26, 2019

@oleksiyp
Hi again, I have experimented a little bit on generating interceptor calls. The idea I followed is to supply interceptor to the class and then call it before executing the rest method (or implement class for interfaces).

I have pushed results so my testbed repo.
Generation of classes is happening here and the end results are described in the test file here.
I haven't yet tried applying those plugins to K/N, as it seems much easier to inspect results on JVM. However, I don't expect implementation to differ much because of common backend.

@oleksiyp
Copy link
Collaborator

Maybe you are right and I am way to strict to make it way too powerful. I searched for compromises, but probably not so extensively. What can I say it may work and even cover 80% of useful cases.

I checked code a bit, but no time to analyze it in details. Can say only if there is really a desire to help the project, I will invest my time as well. For testing, build pipeline and releasing.

@ShikaSD
Copy link

ShikaSD commented Dec 27, 2019

I am surely interested to continue with this, however it would be great to come up with list of features which should be supported, so there will be a certain list to work on.
@oleksiyp Can you please make one based on current state of the library?

Also I have checked possible generation of implementations for mockk<T>() functions and supposedly we can try to generate something after the function has been inlined (so the target type is known). Again, need more time to even understand where to get this inlining info :)

@oleksiyp
Copy link
Collaborator

Hard to tell all the features and actually no big need. Test cases represent features. In javascript I achieve part coverage, in Android instrumented test and JVM full coverage. Javascript version has some flaws and that prevents from making it production ready.

@mikeholler
Copy link

Any updates here? We're looking into using kotlin multiplatform and testing library support is one of our bigger considerations.

@LouisCAD
Copy link

LouisCAD commented Jul 6, 2021

You can use fakes instead of mocks.

@ShikaSD
Copy link

ShikaSD commented Jul 6, 2021

I mostly abandoned idea myself, due to lack of time
I still have a prototype created somewhere, and it kinda works for mocking extendable interfaces - similarly how you can do without any library with fakes.

For everything final/static, mocking is quite annoying to implement as a compiler plugin, just because at the moment it would require recompiling dependencies as well.

@plusmobileapps
Copy link

I have seen two libraries pop up that have provided a way to mock interfaces only in common tests which use kotlin symbol processing called MocKmp and Mockative. I have been a fan of Mockk ever since its inception and was wondering if something similar could be achieved to unlock mocking with interfaces at least with Mockk due to the limitations of Kotlin native?

@only-kuban
Copy link

I support this topic. It is very important to be able to write mocks for Kotlin/Native.

@zhimbura
Copy link

Yes, I have to write mocks for all platform included native too.
Could you say status for this task?

@josegbel
Copy link

I would also love to see mockk working in KMP.

@nathanfallet
Copy link

Any news on MockK for Kotlin Native?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests