Cucumber JVM should provide reuseable API for Options/Parsing/Compiling/Running #1711
Comments
|
What kind of information are you looking for? Directly accessing the |
|
In general I'd like to reuse the parsing glue as well as generation of code snippets. To summarize: Using the eclipse plugin should produce the same errors/output as running against cucumber-cli. |
In general that is going to be a problem because not all glue can be parsed without executing the code. Lamda step definitions for instance require the class that defines them is instantiated. This can be a problem when building the test context is a slow process. But lets disregard Lambda step definitions for now.
Specifically which bits of information are you looking for? And in what format would you like to receive them? I'd rather not open up all objects and let you look inside them -- it would make Cucumber-JVM very hard to evolve -- but we could come up with a JSON format that makes it easier. |
|
I'm sure your using some kind of IDE for development (maybe Idea or Eclipse), so simply think about what would you expect from an IDE to help you write Java code. Of course it is nice to have syntax highlight, but even more helpful are features that provide you with quick-fix and suggestions for autocomplete and you want to get error/warning hints if something is probably wrong. Especially you want to support the "Specification Guys" so the don't need to know what steps are there but can find a list on there hand, automatically generated from projects code. Lets assume I have a feature file and want to add rich edit capabilities for the user.
All the information and features are already available in cucumber-jvm, its just hard to access them because cucumber-jvm is specially designed for the "run once" case. The Lamdastuff is a good example: cucumber-jvm must have some kind of logic to execute and match steps, as a tool author I don't mind HOW cucumber-jvm gather the information (even if its slow) I just don't want to implement the same logic again. So probably I would create an instance of JavaBackend (or any other Interface!) initlize it with some options (e.g. glue packages, classloader) and then I call a method that gives me all steps that would be available the same way as if I would have run cucumber on the cli. So to summarize: I don't wan't new exclusive features for IDE/Tools, just reuse the stuff the cucumber-cli runner will internally already do in a way that does not involve executing the feature with cli and parse the std-out of the process. To archive this I won't mind calling internal API even if that means I has to take care about changes to this internal stuff when I upgrade to a higher version. |
|
I'm not interested in considering the possibility of opening up Cucumber JVM outside of well defined interfaces and API's. However for first three use cases you've described I believe the |
|
Okay so you suggest to use something like this: This would be perfectly fine, i just would need more fain grained control over the "ecexute" stage, since I don't really want to execute (yet) the code, i would need something like:
So do you think the Runtime can be extended like this so the current run() is split into these "Phases" instead of one big "run()"? |
|
If you're looking to do that then you might be better of assembling the run time components yourself like the JUnit and TestNG Runner do. |
|
That might be possible but I would prefer not to replicate all the mechnism. If JUnit an TestNG already do this, wouldn't it be valid to provide reuseable API that
|
laeubi
changed the title
|
We used to have this design in v1 and v2. The The CLI, JUnit TestNG each impose a different structure on the way tests are discovered, organized, filtered and executed. Because each of the runners is interested in low level concepts - discovery, features, scenarios, filtering- this structure does not generalize into a single You might have some luck extracting a generic runner factory of sorts but looking forward to Junit5 I think even that will be done differently by then. |
|
I see you've just changed the title of the issue. That would be a reasonable request but I don't think the current internal implementation is stable yet. |
|
If you're looking to contribute something you could start with the |
|
I have started to investigate using StepDefinition from event, the problem is that I can't get access to the actual method, do you think it would be ok to add a getter for the method that belongs to the step definition? |
|
What do you need the method object for? You don't need the execute method. Would the canonical name/signature and location be sufficient? |
|
Came across this issue and made me toy around with writing a plugin / runner in hopes it helps: import io.cucumber.core.event.*;
import java.util.Map;
import java.util.TreeMap;
public class StepDiscoveryPlugin implements EventListener{
private final Map<String, String> definedSteps = new TreeMap<>();
private final Map<String, Status> finishedSteps = new TreeMap<>();
private final Map<String, Status> finishedCases = new TreeMap<>();
@Override
public void setEventPublisher(EventPublisher publisher) {
publisher.registerHandlerFor(StepDefinedEvent.class, this::handleStepDefinedEvent);
publisher.registerHandlerFor(TestCaseFinished.class, this::handleTestCaseFinished);
publisher.registerHandlerFor(TestStepFinished.class, this::handleTestStepFinished);
}
private void handleStepDefinedEvent(StepDefinedEvent event) {
definedSteps.put(event.getStepDefinition().getLocation(false), event.getStepDefinition().getPattern());
}
private void handleTestStepFinished(TestStepFinished event) {
finishedSteps.put(event.getTestStep().getCodeLocation(), event.getResult().getStatus());
}
private void handleTestCaseFinished(TestCaseFinished event) {
finishedCases.put(event.getTestCase().getUri() + ':' + event.getTestCase().getLine(),
event.getResult().getStatus());
}
public Map<String, String> getDefinedSteps() {
return definedSteps;
}
public Map<String, Status> getFinishedSteps() {
return finishedSteps;
}
public Map<String, Status> getFinishedCases() {
return finishedCases;
}
}Which can then be run through something like: RuntimeOptions runtimeOptions = new RuntimeOptionsBuilder()
.setDryRun()
.build();
StepDiscoveryPlugin stepDiscoveryPlugin = new StepDiscoveryPlugin();
Runtime runtime = Runtime.builder()
.withRuntimeOptions(runtimeOptions)
.withAdditionalPlugins(stepDiscoveryPlugin)
.build();
runtime.run();
Map<String, String> definedSteps = stepDiscoveryPlugin.getDefinedSteps();
Map<String, Status> finishedSteps = stepDiscoveryPlugin.getFinishedSteps();
Map<String, Status> finishedCases = stepDiscoveryPlugin.getFinishedCases();
System.out.println("definedSteps=" + definedSteps);
System.out.println("finishedSteps=" + finishedSteps);
System.out.println("finishedCases=" + finishedCases);Which will print something like: |
|
@laeubi could the above help in the eclipse plugin development? I'm unfamiliar with IDE plugin development constraints, but it would seem to me you have all the parts to make this work, since the Eclipse plugin is already able to run features. Please note that it's enough to run in |
|
@mpkorstanje I need it to show in code editor or jump to the location. I just don't want to parse strings I wan't to use standard java api from objects already there... |
|
This would allow to match source with parsed step definition in most cases. |
|
btw Groovy-Support is also on the todo-list, so if this would work on the groovy-backend as well this would be nice. |
|
I have an interesting, if unrelated issue which might benefit from this sort of API access. I'm currently developing "user simulators" - automated processes that simulate user activity - which for now are being programmed manually. The simulators use Selenium WebDriver (i.e. send clicks, keystrokes, etc., through a surrogate browser instance), and work fairly well as-is. We can very quickly scale up or down the number of users, etc. However, modifying the script the simulators are on requires code changes. Enter Gherkin (and by now you see where I'm going with this). Using Gherkin we could then build a library of "scripts" (i.e. Gherkin features/scenarios) which we could then feed to the underlying engine (i.e. Cucumber) at our "leisure" in order to run the simulated scenarios. I'm well aware that this is only a subset of what an engine like Cucumber can do, and is a bit deviant from the design intent (automated testing). However, this seems like a valid enough use case to wonder: why is this (currently) not possible? I poked around the code a bit to try to figure out if there was some pattern that could be followed to run ONE scenario (or, failing that, a feature file). If that became available we could easily come up with a JMeter plugin that allows features to be defined and executed dynamically during user simulation. As a side note, we're using JMeter since it affords us two important things: thread management and built-in metrics tracking/reporting. So....thoughts? If there already examples of how to do this, then I'd certainly like to see one since my searches have come up empty so far. What I'm (ideally) looking for is something like this: public void initGherkinEngine() {
// initialize all the Cucumber stuff, class introspector/scanner, etc., but NOT feature script loading
this.cucumberEngine = new CucumberEngine();
// ... more configuration ... /
this.cucumberEngine.configure();
}
/** This function would get called time and again for each "Gherkin Sampler" defined in JMeter.
* Thread safety and all that would be handled by our code as well (i.e. if the cucumber engine
* is stateful and can't be reused by different threads). Thus, we initialize the "big" stuff as
* few times as possible, and simply "run" the features as if they were scripts as we go...
*/
public void runFeature(String feature) {
CompiledFeature c = FeatureCompiler.compile(feature);
this.cucumberEngine.execute(c);
}
public void closeGherkinEngine() {
// shut down the Cucumber stuff cleanly, if even necessary
this.cucumberEngine.close();
}Thanks! |
|
You can use the cucumber-main as a blueprint and adjust it to your needs. Another way would be to call the Main via javacode and passing the necessary parameters but since most of the stuff is private in that class you won't have much control over the flow. |
|
@drivera73 it might be worth looking at JUnit 5. In combination with https://junit.org/junit5/docs/current/user-guide/#launcher-api |
|
@laeubi I saw the cucumber-main and tried to use it as a blueprint, but ran into the drawback that each time I would want to run a single feature/story (script), I'd have to (re-)initialize the entire framework from scratch. What I'm looking for is some method or mechanism through which I can initialize the framework in terms of the story step implementations (Given, When, Then, etc.), and then selectively run one story at a time, in whatever order I choose. Imagine a GUI where I can basically define the engine's base step lookup mechanics by some unspecified means, and upon clicking a "start" button I'm presented with a TextArea input where I type in the story I wish to run, along with 5 buttons: load, save, compile (i.e. validate syntax), run, and clear (clear the text area). While that's not the tool I'm looking to implement it embodies everything I would need:
Thoughts? |
|
To clarify: in my use case I'd like to scan the class libraries for the Cucumber step tags only once, since doing that task is likely to be fairly time-consuming. That's what initGherkinEngine() from my above pseudo-code example would do. Basically, any and all initialization of objects that don't need to be built time-and-again when running each feature would happen once per execution thread. Then, once the context is initialized (classes scanned, steps indexed, etc.) I would reuse that context whenever I wish to run a specific story. Then I would simply obtain the story/feature as a string (or somesuch) to be fed into runFeature() for execution. This could be done by many threads at once so the context (which is meant to be re-used) would have to be hardened for that purpose (unless we're using one per thread). Many unrelated stories may end up being run in succession by the same thread. Managing the state within the step implementations is out of scope for this discussion. Finally, when all is done, any and all cleanup would be handled by closeGherkinEngine(). Under this model, it doesn't matter where the features/stories would be loaded from. We'd have to be careful to avoid name/string/syntax collisions between steps since the implementations would be strewn across a myriad of different classes whose (class)path would now be irrelevant, since we're not looking to do automatic matching of a story to its step implementations. Based on what I read in the code so far, there's no way to do this currently exposed... is this impression correct? Adding code beyond Cucumber (and required dependencies) is out of the question - we're trying to be as strict as possible with the size of our binary for reasons that aren't relevant to this conversation. So...any ideas? Thanks! |
|
@drivera73 I fully agree that this would be helpful and I'm partly tried to implement something in this direction but that's a little bit cumbersome because you have to adjust whenever cucumber extends/change... not sure if this was improved in cucumber5 line. |
|
Just as an FYI, I was able to do something close to what I want to do using JBehave. It supports Gherkin as well as its own Gherkin-like syntax (not using this for now) and even though it may not be 100% the syntax that Cucumber supports, it's good enough for my current purposes. I look forward to the day I can support this type of functionality with Cucumber. Cheers! |
|
JUnit 5.7.0 has been released, with notable new addition: https://junit.org/junit5/docs/5.7.0/release-notes/
I believe that was what was needed for this issue to continue; Am I right @laeubi ? |
|
So @mpkorstanje just added the new FilePosition handling in #2121 ; Would that be enough to continue with the cucumber-eclipse plugin @laeubi / @qvdk ? |
|
Hi |
|
@mpkorstanje its been a while but now I'm back to the topic and whonder if in the meanwhile there is some support to supply a set of source/classfiles and let cucumber-jvm scan them for stepdefs like with gherkin described here https://github.com/cucumber/cucumber/tree/master/gherkin ? |
|
Can't do that without executing Cucumber. Step definitions classes currently have to be instantiated (e.g. cucumber-java8) to register step definitions. You may do some static analysis but that's well out side the scope of this project. |
|
That said, you may have some luck with a dry-run in combination with the output from the |
|
@mpkorstanje thanks the dry-run seems to work so far and gives me the step definition information! The only thing I noticed is that I also need to provide a dummy feature file, otherwise cucumber simply do nothing but I assume that's intentional?! |
aslakhellesoy
added
🏦 debt
|
I'd like to give some feedback here. I have now used the dry-run option together with a custom plugin that works really nice. I'm currently using the "Runtime" class as a backend and have written a little wrapper around it mainly to conveniently collect some data (plugins, features) before I start the run and convert some internal data structures: @mpkorstanje already mentioned that this is not really API and seems to have some drawbacks but I found the Runtime usage quite comfortable, flexible and has given good results. So maybe this class can be enhanced to become a public API. I also used the Main/CLI class that also has worked, but I'm missing there the opportunity to pass Plugins/Features as Java objects (as sometimes a feature might only reside in memory or a plugin requires additional parameter that can't be passed as text, and sometime I wan't to access data after the run is over). It also requires me to build some strings to pass parameters and change runtime options what is a bit crazy (I need to make strings out of my object so cucumber can parse the strings into runtime options). |
|
The JUnit5 API is a nice API to copy. If you squint a bit you can already see that the A few important points:
|
|
This API doesn't really facilitate IDEs yet: IDE's would need to be able to:
I think this mostly comes down to how the glue API and the Gherkin parser API are designed. |
Co-authored-by: Renovate Bot <bot@renovateapp.com>
This is related to cucumber/cucumber-eclipse#368
The JavaBackend currently has no way to access SnippetGenerator and TypeRegistry that makes it impossible for IDEs or other code to reuse the information collected by the code.
The text was updated successfully, but these errors were encountered: