Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to inject middleware into rule #7

Closed
mogronalol opened this issue Mar 24, 2016 · 23 comments
Closed

Ability to inject middleware into rule #7

mogronalol opened this issue Mar 24, 2016 · 23 comments

Comments

@mogronalol
Copy link
Contributor

The builder for rule could be modified so you can pass custom middleware into Hoverfly

@tommysitu
Copy link
Member

While injecting middleware is not yet available for hoverfy-java, one work around would be starting up hoverfly with middleware outside JVM, eg. :

hoverctl middleware --binary python --script middleware.py

Then use the external hoverfly in hoverfly-java by setting the useRemoteInstance option:

@ClassRule
    public static HoverflyRule hoverflyRule = HoverflyRule.inSimulationMode(classpath("test-service.json"),
            configs().useRemoteInstance("localhost"));

@jxlampdx
Copy link

@tommysitu, we will give this work around a go. Thanks!

@jxlampdx
Copy link

@tommysitu we are able to invoke the middleware with the above workaround in a JUnit test!

The next progression would be to move this invocation code into a networkHelper class that our app can call and get the synthesized data back. The only caveat is the HoverflyConfig and SimulationSource are both in the io.specto.hoverfly.junit.core package.

We like to have a method like this...

public static String getRoomScheduleSimulation() {

    try {
        HoverflyRule hoverflyRule = HoverflyRule.inCaptureOrSimulationMode(("javaSimulation.json"), HoverflyConfig.configs().useRemoteInstance());

        Process p = Runtime.getRuntime().exec("curl --insecure --proxy http://localhost:8500curl --insecure --proxy http://localhost:8500 https://wpa.steelcase.com/rooms/a7155e79-842e-11e6-a1a5-000d3a321a2e/bookings");

        p.waitFor();
        BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));

        String line = "";
        StringBuffer output = new StringBuffer();

        while((line = reader.readLine()) != null) {
            output.append(line + "\n");

        }

        String wholeResponseString = output.toString();

//or parse the string into Model here...

        return wholeResponseString;

    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}

Any recommendations/thoughts?

Thanks!

@tommysitu
Copy link
Member

@jxlampdx I am trying to understand your problem a bit better. Do you want to simulate a Restful endpoint (https://wpa.steelcase.com/rooms/a7155e79-842e-11e6-a1a5-000d3a321a2e/bookings) in your JUnit test?

Also I am assuming that you are trying to test your application that depends on the synthesised response from that endpoint?

@jxlampdx
Copy link

@tommysitu , yes and yes. I like to be able to simulate and return a middleware injected response from Junit test and also able to have a helper method so that we can call it to get the synthesized data and populate the views...

The problem we are trying to solve is to synthesize response from services that are not yet available from our backend.

@jxlampdx
Copy link

looking for a Gradle Build that is a Compile if it is available? Currently we have testCompile 'io.specto:hoverfly-java:0.3.8'

@tommysitu
Copy link
Member

@jxlampdx It sounds like you want to use the simulation for both your JUnit test, and also inside your application for the time being.

For JUnit test, you only need to add the JUnit Rule to your test class:

    @ClassRule
    public static HoverflyRule hoverflyRule = HoverflyRule.inSimulationMode(classpath("javaSimulation.json"),
            HoverflyConfig.configs().useRemoteInstance());

The JUnit rule would auto-configure JVM proxy and default SSL, therefore you don't need these flags "--insecure --proxy http://localhost:8500". Any particular reason you are invoking curl in java code?

To use the simulation outside JUnit test, for example embedded in your application to be a stub server, you can invoke this when your application starting up:

try(Hoverfly hoverfly = new Hoverfly(HoverflyConfig.configs().useRemoteInstance(), HoverflyMode.SIMULATE)) {

            hoverfly.importSimulation(SimulationSource.classpath("javaSimulation.com"));
            hoverfly.start();
        }

@tommysitu
Copy link
Member

Yes, you can get hoverfly-java with compile / testCompile in Gradle.

@jxlampdx
Copy link

For hoverfly-java to work with both compile/testcompile. I changed the build.graddle to

compile 'io.specto:hoverfly-java:0.3.8 instead of the testCompile 'io.specto:hoverfly-java:0.3.8' .

is that correct?

@jxlampdx
Copy link

this is my dependencies ...

dependencies {
compile fileTree(dir: 'libs', include: ['*.jar'])
androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
exclude group: 'com.android.support', module: 'support-annotations'
})

compile 'io.specto:hoverfly-java:0.3.8'

compile 'com.android.support:appcompat-v7:25.1.0'
compile 'com.android.support:support-v4:25.1.0'
compile 'com.android.support:design:25.1.0'


testCompile 'junit:junit:4.12'

}

and getting this error...

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ':app:transformResourcesWithMergeJavaResForDebug'.

com.android.build.api.transform.TransformException: com.android.builder.packaging.DuplicateFileException: Duplicate files copied in APK META-INF/LICENSE
File1: /Users/jlam1/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.8.4/de3570327cf8d1d4f37920535c51a1f74225a6de/jackson-annotations-2.8.4.jar
File2: /Users/jlam1/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.jaxrs/jackson-jaxrs-base/2.8.4/6c0ceb3c9fed2e225b0cc2a45533574df393f606/jackson-jaxrs-base-2.8.4.jar
File3: /Users/jlam1/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.8.5/b3035f37e674c04dafe36a660c3815cc59f764e2/jackson-databind-2.8.5.jar
File4: /Users/jlam1/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.jaxrs/jackson-jaxrs-json-provider/2.8.4/839366ece31829a19cb15719b2b54a3f9f91148d/jackson-jaxrs-json-provider-2.8.4.jar
File5: /Users/jlam1/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.8.4/d2eec7cf6c4284f7d5f0b1a72dc7cfa9d6bb579d/jackson-module-jaxb-annotations-2.8.4.jar
File6: /Users/jlam1/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.5/60d059f5d2930ccd1ef03535b713fd9f933d1ba7/jackson-core-2.8.5.jar

@tommysitu
Copy link
Member

Using compile is fine, it looks like you need to exclude Jackson-core dependency from hoverfly-java.

@jxlampdx
Copy link

I excluded t 'META-INF/jersey-module-version' and gotten passed the above. but now gotten another runtime error. Tried adding the

compileOptions {
sourceCompatibility JavaVersion.VERSION_1_7
targetCompatibility JavaVersion.VERSION_1_7
}
and also adding
multiDexEnabled true

clean and rebuilt but could get pass this error below. I also tried to remove all the dependencies and added them back one at a time, once I added back

compile 'io.specto:hoverfly-java:0.3.8', will encounter the runtime error above.

Any other ideas? Thanks!

:app:transformClassesWithDexForDebug
AGPBI: {"kind":"error","text":"Error converting bytecode to dex:\nCause: Dex cannot parse version 52 byte code.\nThis is caused by library dependencies that have been compiled using Java 8 or above.\nIf you are using the \u0027java\u0027 gradle plugin in a library submodule add \ntargetCompatibility \u003d \u00271.7\u0027\nsourceCompatibility \u003d \u00271.7\u0027\nto that submodule\u0027s build.gradle file.","sources":[{}],"original":"UNEXPECTED TOP-LEVEL EXCEPTION:\njava.lang.RuntimeException: Exception parsing classes\n\tat com.android.dx.command.dexer.Main.processClass(Main.java:775)\n\tat com.android.dx.command.dexer.Main.processFileBytes(Main.java:741)\n\tat com.android.dx.command.dexer.Main.access$1200(Main.java:88)\n\tat com.android.dx.command.dexer.Main$FileBytesConsumer.processFileBytes(Main.java:1683)\n\tat com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)\n\tat com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)\n\tat com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)\n\tat com.android.dx.command.dexer.Main.processOne(Main.java:695)\n\tat com.android.dx.command.dexer.Main.processAllFiles(Main.java:592)\n\tat com.android.dx.command.dexer.Main.runMultiDex(Main.java:376)\n\tat com.android.dx.command.dexer.Main.run(Main.java:290)\n\tat com.android.builder.internal.compiler.DexWrapper.run(DexWrapper.java:54)\n\tat com.android.builder.core.DexByteCodeConverter.lambda$dexInProcess$0(DexByteCodeConverter.java:173)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: com.android.dx.cf.iface.ParseException: bad class file magic (cafebabe) or version (0034.0000)\n\tat com.android.dx.cf.direct.DirectClassFile.parse0(DirectClassFile.java:476)\n\tat com.android.dx.cf.direct.DirectClassFile.parse(DirectClassFile.java:406)\n\tat com.android.dx.cf.direct.DirectClassFile.parseToInterfacesIfNecessary(DirectClassFile.java:388)\n\tat com.android.dx.cf.direct.DirectClassFile.getMagic(DirectClassFile.java:251)\n\tat com.android.dx.command.dexer.Main.parseClass(Main.java:787)\n\tat com.android.dx.command.dexer.Main.access$1600(Main.java:88)\n\tat com.android.dx.command.dexer.Main$ClassParserTask.call(Main.java:1722)\n\tat com.android.dx.command.dexer.Main.processClass(Main.java:773)\n\t... 16 more\n","tool":"Dex"}
AGPBI: {"kind":"error","text":"1 error; aborting","sources":[{}]}

FAILED

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ':app:transformClassesWithDexForDebug'.

com.android.build.api.transform.TransformException: java.lang.RuntimeException: java.lang.RuntimeException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: Return code 1 for dex process

@tommysitu
Copy link
Member

That's because hoverfly-java requires java 8 to compile.

@jxlampdx
Copy link

I see. now that I stepped back and look, if we have the remote instance running as a proxy, we don't actually need the integration in the Android app to invoke it. As long as we inject the middleware code before we call the service in the Android app, we should be good.

Thanks! @tommysitu

@tommysitu
Copy link
Member

No problem @jxlampdx. We are always looking for interesting use cases of Hoverfly. Could you be contacted via email? We are quite keen to find out how you are using hoverfly for mobile development. Thanks.

@JohnFDavenport
Copy link
Contributor

If it helps we have a contact form near the foot of the http://specto.io/ page so no one needs to disclose their email here.

@jxlampdx
Copy link

@JohnFDavenport and @tommysitu, I filled out the contact form. Thanks again

@tommysitu tommysitu removed this from the 0.4.0 milestone Mar 6, 2017
@benjih
Copy link
Contributor

benjih commented Mar 15, 2017

Hey,

I have been thinking about middleware a lot recently and how it should be used.

There are currently two ways of executing middleware in Hoverfly. The first way is through a process that is started for each request. This has performance issues that occur as you start increase the load of Hoverfly. The reason for this is due to spinning up a new process for each request in flight. The result can be hundreds of child processes all spawned from Hoverfly.

The other way, (which is currently undocumented as of the time of writing this) is running middleware over HTTP. This allows you to turn your middleware into a HTTP server (or maybe a serverless function a la AWS Lambda). Using this approach means that you generally only need one process to run the HTTP server which can be reused by Hoverfly, reducing the performance overhead with process creation.

I have been thinking for a while that maybe the default way people should be implementing middleware is as HTTP servers. I believe its simpler and it is definitely more performant.

With the middleware context out of the way, I think implementing middleware as a HTTP server in hoverfly-java could be much easier than worrying about middleware processes. In golang, it is very easy to produce HttpHandler functions that match a HTTPHandler interface to quickly produce HTTP servers. Maybe hoverfly-java users could define their middleware in their tests as a MiddlewareFunction that is then passed to hoverfly-java which then serves it and provides the address to the running instance of Hoverfly?

Your opinions @mogronalol and @tommysitu?

@mogronalol
Copy link
Contributor Author

@benjih Yes I'm in agreement. We can implement a webserver middleware which starts in the same process as the test, as part of the Java code. This actually frees Java developers up to do whatever they want with the data.

@JohnFDavenport
Copy link
Contributor

Seems like a great idea @benjih @mogronalol but I have some questions. (1) Would this be generic enough to be used in other languages? (2) Throttling: You could still flood the webserver with thousands of requests during a performance test. At the moment responsibility is being passed from the very efficient multi-threading Go process to a poorly scaling Linux process. The issues start when the tester wants to do something long-running like add a delay. How much better will this solution be? A webserver is unlikely to be more efficient than Go and will hit the same issues, albeit at a higher load. If the webserver calls out to a database will it have to manage a connection pool? It may be better to throttle at the Hoverfly end of things using some form of internal queue to maintain order. That might allow us to keep the webserver relatively simple. (3) When. After Hoverfly 1.0 please.

@mogronalol
Copy link
Contributor Author

mogronalol commented Mar 20, 2017

  1. This example would be webserver in Java as part of the same code / process as the test
  2. We already support webserver mode in Hoverfly. This is just a suggestion on how it could be used with Hoverfly Java. It's already faster than binary middleware because you only have to start the process once. Also, contrary to popular belief the JVM is still very fast and scalable, even when doing things in parallel. How throttling is implemented is up to the developer. Hoverfly already supports delays, as I'd probably just use that.

@tommysitu
Copy link
Member

tommysitu commented Mar 21, 2017

@JohnFDavenport It won't be generic enough for other languages. We will need to start up embedded web server in JUnit tests using lightweight web framework such as Spark: https://github.com/perwendel/spark/

For simulating delay and network latency, it is better to have native support from hoverfly, and avoid the use of middleware whenever we could.

@tommysitu
Copy link
Member

Support for using local middleware is available, thanks to @dipak-pawar!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants