New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better experience when running tests with IntelliJ/WebStorm #18

Open
jan-molak opened this Issue Jan 23, 2017 · 16 comments

Comments

3 participants
@jan-molak
Owner

jan-molak commented Jan 23, 2017

To provide a better developer experience when running tests with IntelliJ and WebStorm, Serenity/JS should report additional information to the Protractor test runner:

  • Test execution time, via a durationMillis property on the scenarioInfo object

It seems like it would be cool to also report:

  • Top-level steps executed (Given/When/Then), so that the test execution tree can provide more information (YouTrack)
  • "ignored" (mocha) and "pending" (cucumber) steps - currently Protractor only recognises passes and failures, but we could augment the testInfo object with some additional information. @segrey?
  • I'm not sure what would enable going from a specific test result in the test execution tree to the actual file? Probably IntelliJ would need to know about the file path somehow? @segrey?
@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey Jan 23, 2017

@jan-molak Thank you for filing this! These features are missing in current protractor+cucumber+IntelliJ integration. Would be great to add them.

Top-level steps executed (Given/When/Then), so that the test execution tree can provide more information

Would be great!

"ignored" (mocha) and "pending" (cucumber) steps - currently Protractor only recognises passes and failures, but we could augment the testInfo object with some additional information.

Ideally, protractor API should be extended here, but a workaround we could augment the testInfo object as long as protractor console runner works with this augmentation.
Currently, the augmented testInfo object could be emitted as 'passed' or 'failed' event. Let's for certainty, consider the 'passed' event object. If testInfo.category is undefined, previously released IntelliJ builds will skip this event, i.e. behave correctly. (BTW, you can find IntelliJ protractor integration code inside WebStorm-installation-dir/plugins/JavaScriptLanguage/helpers/protractor-intellij/.)
For example, testInfo could look like

{
  "status": "disabled or focused",
  "originalTestInfo": {
    "category": "my category",
    "name": "my name"
  }
}

I'm not sure what would enable going from a specific test result in the test execution tree to the actual file? Probably IntelliJ would need to know about the file path somehow?

Here is how it works now. For javascript files, it needs just full test name (i.e. all parent spec names + test name). A file path would be great as long as it's a real source file path (not a compiled file).
For cucumber, step definitions and step declarations are not handled at all. Thus, it'd be great to augment the testInfo object with complete info in this case, like

{
    "category": "my category",
    "name": "my name",
    "sourceLocation": {
        "filePath": "/path/to/file",
        "line": "<one based line number>"
        "column": "<one based column number>"
    }
}

segrey commented Jan 23, 2017

@jan-molak Thank you for filing this! These features are missing in current protractor+cucumber+IntelliJ integration. Would be great to add them.

Top-level steps executed (Given/When/Then), so that the test execution tree can provide more information

Would be great!

"ignored" (mocha) and "pending" (cucumber) steps - currently Protractor only recognises passes and failures, but we could augment the testInfo object with some additional information.

Ideally, protractor API should be extended here, but a workaround we could augment the testInfo object as long as protractor console runner works with this augmentation.
Currently, the augmented testInfo object could be emitted as 'passed' or 'failed' event. Let's for certainty, consider the 'passed' event object. If testInfo.category is undefined, previously released IntelliJ builds will skip this event, i.e. behave correctly. (BTW, you can find IntelliJ protractor integration code inside WebStorm-installation-dir/plugins/JavaScriptLanguage/helpers/protractor-intellij/.)
For example, testInfo could look like

{
  "status": "disabled or focused",
  "originalTestInfo": {
    "category": "my category",
    "name": "my name"
  }
}

I'm not sure what would enable going from a specific test result in the test execution tree to the actual file? Probably IntelliJ would need to know about the file path somehow?

Here is how it works now. For javascript files, it needs just full test name (i.e. all parent spec names + test name). A file path would be great as long as it's a real source file path (not a compiled file).
For cucumber, step definitions and step declarations are not handled at all. Thus, it'd be great to augment the testInfo object with complete info in this case, like

{
    "category": "my category",
    "name": "my name",
    "sourceLocation": {
        "filePath": "/path/to/file",
        "line": "<one based line number>"
        "column": "<one based column number>"
    }
}
@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Jan 23, 2017

Owner

Top-level steps executed (Given/When/Then), so that the test execution tree can provide more information

Would be great!

I wonder if we could do something interesting with nesting the test tree nodes here too?
Serenity/JS strongly encourages using the Screenplay Pattern, which places focus on task composition.

For example, a test scenario where say Adding a product to the basket affects the total price could be modelled as follows:

Feature: Basket Management                                          |  Cucumber-land
                                                                    |
  Scenario: Adding a product to the basket affects the total price  |
                                                                    |
    Given James wants to buy a product on amazon.com                |
                                                                    |  ---
    - James opens http://amazon.com                                 |  Serenity/JS-land
    - James signs in with his credentials                           |
      - James click the Sign in button                              |
      - James enters his credentials                                |
        - James enters his email address                            |
        - James enters his password                                 |
        - James clicks the Sign in button                           |

so that it's reported along the lines of:

screen shot 2017-01-23 at 18 32 30

I think a mechanism used to report on Serenity/JS tasks could be similar to the one used to report on Cucumber's Given/When/Then steps, but I guess the tricky thing would be to get the ordering right because IntelliJ sorts the tree to show nodes with leaves before other leaves at the same level.

Also, we'd need to make a distinction when reporting a node vs a leaf I suppose?

And most importantly, Protractor might need to enable some sort of an event-bus for intra-plugin communication (could be outside of scope for Protractor; maybe IntelliJ should listen on the Serenity/JS event bus?)


"ignored" (mocha) and "pending" (cucumber) steps - currently, Protractor only recognises passes and failures, but we could augment the testInfo object with some additional information.

Ideally, protractor API should be extended here,

Agreed, filed at angular/protractor#3997

but a workaround we could augment the testInfo object as long as protractor console runner works with this augmentation.

I think that as long as we keep the original properties untouched and only augment the testInfo with some additional ones, it should be fine (?). Protractor doesn't seem to be doing any validation, but other plugins will most certainly expect the original properties to be in place.

Currently, the augmented testInfo object could be emitted as 'passed' or 'failed' event. Let's for certainty, consider the 'passed' event object. If testInfo.category is undefined, previously released IntelliJ builds will skip this event, i.e. behave correctly. (BTW, you can find IntelliJ protractor integration code inside WebStorm-installation-dir/plugins/JavaScriptLanguage/helpers/protractor-intellij/.)
For example, testInfo could look like

{
  "status": "disabled or focused",
  "originalTestInfo": {
    "category": "my category",
    "name": "my name"
  }
}

To keep the object compatible with what other Protractor plugins would expect I'd probably rather go for something along the lines of:

  {
    category: 'Basket Management', // feature name (cucumber) or describe (mocha)
    name: 'Adding a product (..)', // scenario (cucumber) or it(mocha),
    meta: {                        // additional, IDE-specific stuff:
      status: 'disabled',
      source: {
        filePath: '/absolute/path/to/file',
        line: '<one based line number>',
        column: '<one based column number>'
      }
    }

This way we can emit whatever metadata is needed and remain good neighbours ;-)

(BTW, you can find IntelliJ protractor integration code inside WebStorm-installation-dir/plugins/JavaScriptLanguage/helpers/protractor-intellij/.)

Thanks, I'll have a look.

Owner

jan-molak commented Jan 23, 2017

Top-level steps executed (Given/When/Then), so that the test execution tree can provide more information

Would be great!

I wonder if we could do something interesting with nesting the test tree nodes here too?
Serenity/JS strongly encourages using the Screenplay Pattern, which places focus on task composition.

For example, a test scenario where say Adding a product to the basket affects the total price could be modelled as follows:

Feature: Basket Management                                          |  Cucumber-land
                                                                    |
  Scenario: Adding a product to the basket affects the total price  |
                                                                    |
    Given James wants to buy a product on amazon.com                |
                                                                    |  ---
    - James opens http://amazon.com                                 |  Serenity/JS-land
    - James signs in with his credentials                           |
      - James click the Sign in button                              |
      - James enters his credentials                                |
        - James enters his email address                            |
        - James enters his password                                 |
        - James clicks the Sign in button                           |

so that it's reported along the lines of:

screen shot 2017-01-23 at 18 32 30

I think a mechanism used to report on Serenity/JS tasks could be similar to the one used to report on Cucumber's Given/When/Then steps, but I guess the tricky thing would be to get the ordering right because IntelliJ sorts the tree to show nodes with leaves before other leaves at the same level.

Also, we'd need to make a distinction when reporting a node vs a leaf I suppose?

And most importantly, Protractor might need to enable some sort of an event-bus for intra-plugin communication (could be outside of scope for Protractor; maybe IntelliJ should listen on the Serenity/JS event bus?)


"ignored" (mocha) and "pending" (cucumber) steps - currently, Protractor only recognises passes and failures, but we could augment the testInfo object with some additional information.

Ideally, protractor API should be extended here,

Agreed, filed at angular/protractor#3997

but a workaround we could augment the testInfo object as long as protractor console runner works with this augmentation.

I think that as long as we keep the original properties untouched and only augment the testInfo with some additional ones, it should be fine (?). Protractor doesn't seem to be doing any validation, but other plugins will most certainly expect the original properties to be in place.

Currently, the augmented testInfo object could be emitted as 'passed' or 'failed' event. Let's for certainty, consider the 'passed' event object. If testInfo.category is undefined, previously released IntelliJ builds will skip this event, i.e. behave correctly. (BTW, you can find IntelliJ protractor integration code inside WebStorm-installation-dir/plugins/JavaScriptLanguage/helpers/protractor-intellij/.)
For example, testInfo could look like

{
  "status": "disabled or focused",
  "originalTestInfo": {
    "category": "my category",
    "name": "my name"
  }
}

To keep the object compatible with what other Protractor plugins would expect I'd probably rather go for something along the lines of:

  {
    category: 'Basket Management', // feature name (cucumber) or describe (mocha)
    name: 'Adding a product (..)', // scenario (cucumber) or it(mocha),
    meta: {                        // additional, IDE-specific stuff:
      status: 'disabled',
      source: {
        filePath: '/absolute/path/to/file',
        line: '<one based line number>',
        column: '<one based column number>'
      }
    }

This way we can emit whatever metadata is needed and remain good neighbours ;-)

(BTW, you can find IntelliJ protractor integration code inside WebStorm-installation-dir/plugins/JavaScriptLanguage/helpers/protractor-intellij/.)

Thanks, I'll have a look.

@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey Jan 23, 2017

I think a mechanism used to report on Serenity/JS tasks could be similar to the one used to report on Cucumber's Given/When/Then steps, but I guess the tricky thing would be to get the ordering right because IntelliJ sorts the tree to show nodes with leaves before other leaves at the same level.

Strange, the tree is unsorted for me as expected. Maybe you have "Sort Alphabetically" and/or "Sort By Duration" toggled? What's your IDE version (Help | About)?
test-result-tree

And most importantly, Protractor might need to enable some sort of an event-bus for intra-plugin communication (could be outside of scope for Protractor; maybe IntelliJ should listen on the Serenity/JS event bus?)

Yes, IntelliJ could listen for Serenity/JS events. In fact, it already does so for jasmine (plugins/JavaScriptLanguage/src/helpers/protractor-intellij/lib/protractor-intellij-jasmine-reporter.js), because Protractor API is too limited now. Maybe it's time to suggest some Protractor API improvements to have a common standardized API, so in future other tools could work with Protractor API only. Anyway, it's fine to use Serenity API for now, I think.

This way we can emit whatever metadata is needed and remain good neighbours ;-)

Not sure I understand: if a compatible object is emitted as 'passed' event for disabled/focused tests, wouldn't it increase number of passed tests in test results of protractor console runner? As for IntelliJ protractor runner, sure, it would handle it correctly, thanks to testInfo.meta.status.

segrey commented Jan 23, 2017

I think a mechanism used to report on Serenity/JS tasks could be similar to the one used to report on Cucumber's Given/When/Then steps, but I guess the tricky thing would be to get the ordering right because IntelliJ sorts the tree to show nodes with leaves before other leaves at the same level.

Strange, the tree is unsorted for me as expected. Maybe you have "Sort Alphabetically" and/or "Sort By Duration" toggled? What's your IDE version (Help | About)?
test-result-tree

And most importantly, Protractor might need to enable some sort of an event-bus for intra-plugin communication (could be outside of scope for Protractor; maybe IntelliJ should listen on the Serenity/JS event bus?)

Yes, IntelliJ could listen for Serenity/JS events. In fact, it already does so for jasmine (plugins/JavaScriptLanguage/src/helpers/protractor-intellij/lib/protractor-intellij-jasmine-reporter.js), because Protractor API is too limited now. Maybe it's time to suggest some Protractor API improvements to have a common standardized API, so in future other tools could work with Protractor API only. Anyway, it's fine to use Serenity API for now, I think.

This way we can emit whatever metadata is needed and remain good neighbours ;-)

Not sure I understand: if a compatible object is emitted as 'passed' event for disabled/focused tests, wouldn't it increase number of passed tests in test results of protractor console runner? As for IntelliJ protractor runner, sure, it would handle it correctly, thanks to testInfo.meta.status.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Jan 23, 2017

Owner

Tree sorting

I'm on:

IntelliJ IDEA 2016.3.2
Build #IU-163.10154.41, built on December 21, 2016
JRE: 1.8.0_112-release-408-b6 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

I used a regular mocha test runner to demonstrate the concept, rather than the protractor runner, so the issue might be unrelated to what we're discussing. Anyway, here's the sample case I used:

describe('Basket Management', () => {
    describe('Adding a product to the basket affects the total price', () => {
        describe('Given James wants to buy a product on amazon.com', () => {

            it('James opens http://amazon.com');

            describe('James signs in with his credentials', () => {

                it('James clicks the Sign in button');
                
                describe('James enters his credentials', () => {
                    it('James enters his email address');
                    it('James enters his password');
                    it('James clicks the Sign in button');
                });

                it('James should see that he\'s logged in');
            });
        });
    });
});

And the test tree (with no sorting I'm aware of):

screen shot 2017-01-23 at 23 08 45

Yes, IntelliJ could listen for Serenity/JS events. In fact, it already does so for jasmine (plugins/JavaScriptLanguage/src/helpers/protractor-intellij/lib/protractor-intellij-jasmine-reporter.js), because Protractor API is too limited now. Maybe it's time to suggest some Protractor API improvements to have a common standardized API, so in future other tools could work with Protractor API only.

I had a look at the protractor-intellij-jasmine-reporter.js. Thanks for the pointer! I see you're using teamcity messaging protocol, clever :-)

Could you please advise what is the status of the protractor-intellij-tree.js script? This seems to be the core communication module...

Would you (and JetBrains) be happy to release it as an open-source npm module? This way Serenity/JS could ship with an IntelliJ plugin that depends on it to enable the communication with the IDE.

This way we can emit whatever metadata is needed and remain good neighbours ;-)

Not sure I understand: if a compatible object is emitted as 'passed' event for disabled/focused tests, wouldn't it increase number of passed tests in test results of protractor console runner? As for IntelliJ protractor runner, sure, it would handle it correctly, thanks to testInfo.meta.status.

Yeah, because of this reason it would need to be limited to only scenarioInfo events. OK, let's scratch that, bad idea. I think plugging into the Serenity/JS messaging bus is going to be cleaner.

Owner

jan-molak commented Jan 23, 2017

Tree sorting

I'm on:

IntelliJ IDEA 2016.3.2
Build #IU-163.10154.41, built on December 21, 2016
JRE: 1.8.0_112-release-408-b6 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

I used a regular mocha test runner to demonstrate the concept, rather than the protractor runner, so the issue might be unrelated to what we're discussing. Anyway, here's the sample case I used:

describe('Basket Management', () => {
    describe('Adding a product to the basket affects the total price', () => {
        describe('Given James wants to buy a product on amazon.com', () => {

            it('James opens http://amazon.com');

            describe('James signs in with his credentials', () => {

                it('James clicks the Sign in button');
                
                describe('James enters his credentials', () => {
                    it('James enters his email address');
                    it('James enters his password');
                    it('James clicks the Sign in button');
                });

                it('James should see that he\'s logged in');
            });
        });
    });
});

And the test tree (with no sorting I'm aware of):

screen shot 2017-01-23 at 23 08 45

Yes, IntelliJ could listen for Serenity/JS events. In fact, it already does so for jasmine (plugins/JavaScriptLanguage/src/helpers/protractor-intellij/lib/protractor-intellij-jasmine-reporter.js), because Protractor API is too limited now. Maybe it's time to suggest some Protractor API improvements to have a common standardized API, so in future other tools could work with Protractor API only.

I had a look at the protractor-intellij-jasmine-reporter.js. Thanks for the pointer! I see you're using teamcity messaging protocol, clever :-)

Could you please advise what is the status of the protractor-intellij-tree.js script? This seems to be the core communication module...

Would you (and JetBrains) be happy to release it as an open-source npm module? This way Serenity/JS could ship with an IntelliJ plugin that depends on it to enable the communication with the IDE.

This way we can emit whatever metadata is needed and remain good neighbours ;-)

Not sure I understand: if a compatible object is emitted as 'passed' event for disabled/focused tests, wouldn't it increase number of passed tests in test results of protractor console runner? As for IntelliJ protractor runner, sure, it would handle it correctly, thanks to testInfo.meta.status.

Yeah, because of this reason it would need to be limited to only scenarioInfo events. OK, let's scratch that, bad idea. I think plugging into the Serenity/JS messaging bus is going to be cleaner.

@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey Jan 24, 2017

I used a regular mocha test runner to demonstrate the concept

Thanks, the issue has been reproduced for me when running mocha tests (mochajs/mocha#2684). Yes, seems to be not related to protractor test runner.

Could you please advise what is the status of the protractor-intellij-tree.js script? This seems to be the core communication module...

Yes, it's more a core communication module. It's based on a modified teamcity messages protocol, not https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity. It's done so to improve communication with IDE.

Would you (and JetBrains) be happy to release it as an open-source npm module? This way Serenity/JS could ship with an IntelliJ plugin that depends on it to enable the communication with the IDE.

Sorry, not sure, because releasing an open-source npm module would require us to maintain backward compatibility with IDE. Another source of problems is to maintain backward compatibility between two versions of this module (one version is bundled with IDE and the other one is npm-installed in serenity module). Although you have a point here, I'd rather keep serenity-specific code in protractor-intellij/lib/protractor-intellij-serenity-reporter.js for a start.

segrey commented Jan 24, 2017

I used a regular mocha test runner to demonstrate the concept

Thanks, the issue has been reproduced for me when running mocha tests (mochajs/mocha#2684). Yes, seems to be not related to protractor test runner.

Could you please advise what is the status of the protractor-intellij-tree.js script? This seems to be the core communication module...

Yes, it's more a core communication module. It's based on a modified teamcity messages protocol, not https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity. It's done so to improve communication with IDE.

Would you (and JetBrains) be happy to release it as an open-source npm module? This way Serenity/JS could ship with an IntelliJ plugin that depends on it to enable the communication with the IDE.

Sorry, not sure, because releasing an open-source npm module would require us to maintain backward compatibility with IDE. Another source of problems is to maintain backward compatibility between two versions of this module (one version is bundled with IDE and the other one is npm-installed in serenity module). Although you have a point here, I'd rather keep serenity-specific code in protractor-intellij/lib/protractor-intellij-serenity-reporter.js for a start.

jan-molak added a commit that referenced this issue Feb 11, 2017

feat(serenity-protractor): Support for Protractor 5.1.x
This change adds support for using Serenity/JS with Protractor 5.0.0, 5.1.0 and 5.1.1:
- `ProtractorNotifier` deals with Protractor 5.1.0 requirement for test frameworks to invoke `runner.afterEach` after every test,
so that the browser is correctly restarted between the tests. See angular/protractor#4087
- `TestFrameworkDetector` registers `mochaOpts` and `cucumberOpts` as valid command line options,
benefiting from angular/protractor#3994 and dealing with a bug introduced in Protractor 5.0.0, which prevents command line arguments
from being passed on to custom test frameworks. See angular/protractor#3978
- Updated the examples and documentation to work with Protractor 5.1.x
- Enables #18

Related issues: angular/protractor#3978 angular/protractor#3994 angular/protractor#4087 #18

jan-molak added a commit that referenced this issue Feb 12, 2017

refactor(core): Domain classes represent the domain concepts better
- `Performable` is really an `Activity` that an `Actor` can perform, therefore it should be called as such.
- What used to be an `Activity` is now a `RecordedActivity`. `RecordedActivity` is just a "tiny type", that captures the name of the real `Activity` and works alongside with `Outcome` to capture the result and other meta data related to the performance.
- To make the model consistent, `Scene` is now a `RecordedScene`
- To help distinguish `RecordedActivity` related to a `Task` and `Interaction`, the `@step()` annotation can now take an optional second arugment `ActivityType.Task` or `ActivityType.Interaction`. A `@step` that doesn't specify its type is considerd to be of type `Task` by default as 90% of time you'll be writing Tasks not Interactions.
- The distinction between a `RecordedTask` and `RecordedInteraction` will help to make the configuration of the `Photographer` more fine-grained, so that we could for example tell it to only capture screenshots of interactions rather than all the tasks. This will also help to enable the granularity needed for #18.
- `See` and `CompareNotes` are now `Interactions` rather than generic `Activities` (pretending to be `Tasks`), as performing an assertion is also an interaction with the system. No need for special treatment of those two classes.
- The `Interaction` interface is also corrected to allow for `Actors` that `UseAbilities` but also `AnswerQuestions`

Those changes should not affect the consumers of the Serenity/JS APIs and will help tackle the tech debt before it spreads ;-)

Enables #18, #22, #23
@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Apr 15, 2017

Owner

Hey @segrey,

I think we can start with reporting top-level step information (Cucumber's given/when/then) before reporting the low-level stuff such as Serenity/JS tasks and interactions - that requires a bit more integration work, so I think I'd rather do it in two stages.

Does the below structure still reflect what you'd like to receive (you as in "IntelliJ/WebStorm" ;-) )?

{
    "category": "my category",
    "name": "my name",
    "sourceLocation": {
        "filePath": "/path/to/file",
        "line": "<one based line number>"
        "column": "<one based column number>"
    }
}

Or would you rather go with something along the lines of you've proposed in protractor-cucumber-framework/protractor-cucumber-framework#81:

{
    "category": "scenario name",
    "name": "step name",
    "sourceLocation": {
        "category": { filePath, line, column }         // scenario details
        "name": { filePath, line, column}              // step details
    }
}

I'm looking for a structure that could support cucumber 1/cucumber 2 as well as mocha.

What information does IntelliJ require to do the navigation?

Owner

jan-molak commented Apr 15, 2017

Hey @segrey,

I think we can start with reporting top-level step information (Cucumber's given/when/then) before reporting the low-level stuff such as Serenity/JS tasks and interactions - that requires a bit more integration work, so I think I'd rather do it in two stages.

Does the below structure still reflect what you'd like to receive (you as in "IntelliJ/WebStorm" ;-) )?

{
    "category": "my category",
    "name": "my name",
    "sourceLocation": {
        "filePath": "/path/to/file",
        "line": "<one based line number>"
        "column": "<one based column number>"
    }
}

Or would you rather go with something along the lines of you've proposed in protractor-cucumber-framework/protractor-cucumber-framework#81:

{
    "category": "scenario name",
    "name": "step name",
    "sourceLocation": {
        "category": { filePath, line, column }         // scenario details
        "name": { filePath, line, column}              // step details
    }
}

I'm looking for a structure that could support cucumber 1/cucumber 2 as well as mocha.

What information does IntelliJ require to do the navigation?

@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey Apr 16, 2017

Hey Jan!

Does the below structure still reflect what you'd like to receive (you as in "IntelliJ/WebStorm" ;-) )?

No, it doesn't. Unfortunately, this structure doesn't allow to navigate from scenario nodes.
I think something like will work for scenario and steps nodes.

{
  "category": "scenario name",
  "name": "step name",
  "sourceLocation": {
    "category": { // scenario details
      "filePath": "/path/to/file",
      "line": "<one based line number>",
      "column": "<one based column number>"
    },
    "name": { // step details
      "filePath": "/path/to/file",
      "line": "<one based line number>",
      "column": "<one based column number>"
    }
  }
}

This is just a possible extension based on structure of protractor testInfo object (https://github.com/angular/protractor/blob/master/lib/frameworks/README.md#requirements). If you need a more deeper test tree, let's discuss it in a separate issue.

segrey commented Apr 16, 2017

Hey Jan!

Does the below structure still reflect what you'd like to receive (you as in "IntelliJ/WebStorm" ;-) )?

No, it doesn't. Unfortunately, this structure doesn't allow to navigate from scenario nodes.
I think something like will work for scenario and steps nodes.

{
  "category": "scenario name",
  "name": "step name",
  "sourceLocation": {
    "category": { // scenario details
      "filePath": "/path/to/file",
      "line": "<one based line number>",
      "column": "<one based column number>"
    },
    "name": { // step details
      "filePath": "/path/to/file",
      "line": "<one based line number>",
      "column": "<one based column number>"
    }
  }
}

This is just a possible extension based on structure of protractor testInfo object (https://github.com/angular/protractor/blob/master/lib/frameworks/README.md#requirements). If you need a more deeper test tree, let's discuss it in a separate issue.

@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey Apr 17, 2017

@jan-molak Sorry, missed the question.

What information does IntelliJ require to do the navigation?

Currently, WebStorm (2017.1.1 as of now) doesn't support any provided navigation details. We need to settle down with how this navigation details would extend testInfo object. A simple possible extension is described in my previous comment. Another alternative that would allow to describe deeper test tree structures could be like this:

{
  // "category" is kept for backward compatibility with Protractor API,
  //  not used by IDE if "parent" property is provided
  "category": "a joined name of all ancestors of this step",
  "name": "step name",
  "sourceLocation": { // step details
    "filePath": "/path/to/file",
    "line": "<one based line number>",
    "column": "<one based column number>"
  },
  "parent": {
    "name": "<name of the step's parent scenario>",
    "sourceLocation": {
      "filePath": "/path/to/file",
      "line": "<one based line number>",
      "column": "<one based column number>"
    },
    "parent": {
      "name": "<name of the step's grandparent scenario>",
      "sourceLocation": {
        "filePath": "/path/to/file",
        "line": "<one based line number>",
        "column": "<one based column number>"
      },
      "parent": {
        ...
        "parent": {
          "name": "name of the step's top level scenario ancestor>",
          "sourceLocation": {
            "filePath": "/path/to/file",
            "line": "<one based line number>",
            "column": "<one based column number>"
          }
        }
      }
    }
  }
}

What do you think?

segrey commented Apr 17, 2017

@jan-molak Sorry, missed the question.

What information does IntelliJ require to do the navigation?

Currently, WebStorm (2017.1.1 as of now) doesn't support any provided navigation details. We need to settle down with how this navigation details would extend testInfo object. A simple possible extension is described in my previous comment. Another alternative that would allow to describe deeper test tree structures could be like this:

{
  // "category" is kept for backward compatibility with Protractor API,
  //  not used by IDE if "parent" property is provided
  "category": "a joined name of all ancestors of this step",
  "name": "step name",
  "sourceLocation": { // step details
    "filePath": "/path/to/file",
    "line": "<one based line number>",
    "column": "<one based column number>"
  },
  "parent": {
    "name": "<name of the step's parent scenario>",
    "sourceLocation": {
      "filePath": "/path/to/file",
      "line": "<one based line number>",
      "column": "<one based column number>"
    },
    "parent": {
      "name": "<name of the step's grandparent scenario>",
      "sourceLocation": {
        "filePath": "/path/to/file",
        "line": "<one based line number>",
        "column": "<one based column number>"
      },
      "parent": {
        ...
        "parent": {
          "name": "name of the step's top level scenario ancestor>",
          "sourceLocation": {
            "filePath": "/path/to/file",
            "line": "<one based line number>",
            "column": "<one based column number>"
          }
        }
      }
    }
  }
}

What do you think?

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Apr 18, 2017

Owner

Protractor requires the frameworks to report the results at a scenario level rather than a step level, so it feels like emitting the testInfo object after every step might be abusing the protocol a bit. After all, there may be other plugins/frameworks that rely on the event being emitted after the scenario.

If we want to comply with the Protractor protocol, it means that given the following scenario:

# /users/jan/demo/features/discount_codes.feature

Feature: Discount codes for selected products

  In order to increase the sales of the products I want to destock
  As a sales manager
  I want to enable customers to buy them at a discounted rate

  Scenario: Using a discount code received from the online store

    Given that I have received the SMARTPHONE_15 discount code
     When I shop for a smartphone
      And I apply my discount code at checkout
     Then I should see that the total price is reduced by 15%

we should emit the following object, together with testPass/testFail, when the scenario is finished:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
}

Adding the durationMillis is not controversial, so we'd have:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
  "durationMillis": 547
}

Even if we don't use the testInfo to report the steps of the scenario (I think that using the beefed up version of the TeamCity protocol you mentioned earlier is better suited for that), we still need to provide the IDE with the scenario and feature details.

I think that the below structure could do the trick:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
  "durationMillis": 547,
  "source": {
    "category": {
      "path": "/users/jan/demo/features/discount_codes.feature",
      "line": 3,
      "column": 1
    },
    "name": {
      "path": "/users/jan/demo/features/discount_codes.feature",
      "line": 9,
      "column": 3
    },
  }
}

It can be used to support both mocha and cucumber and should help the IDE do the navigation work.

The next step will be to use the TC protocol to report on given/when/then steps as well as tasks and interactions.

I think this should be a fairly clean approach. Would that work for you @segrey?

J

Owner

jan-molak commented Apr 18, 2017

Protractor requires the frameworks to report the results at a scenario level rather than a step level, so it feels like emitting the testInfo object after every step might be abusing the protocol a bit. After all, there may be other plugins/frameworks that rely on the event being emitted after the scenario.

If we want to comply with the Protractor protocol, it means that given the following scenario:

# /users/jan/demo/features/discount_codes.feature

Feature: Discount codes for selected products

  In order to increase the sales of the products I want to destock
  As a sales manager
  I want to enable customers to buy them at a discounted rate

  Scenario: Using a discount code received from the online store

    Given that I have received the SMARTPHONE_15 discount code
     When I shop for a smartphone
      And I apply my discount code at checkout
     Then I should see that the total price is reduced by 15%

we should emit the following object, together with testPass/testFail, when the scenario is finished:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
}

Adding the durationMillis is not controversial, so we'd have:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
  "durationMillis": 547
}

Even if we don't use the testInfo to report the steps of the scenario (I think that using the beefed up version of the TeamCity protocol you mentioned earlier is better suited for that), we still need to provide the IDE with the scenario and feature details.

I think that the below structure could do the trick:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
  "durationMillis": 547,
  "source": {
    "category": {
      "path": "/users/jan/demo/features/discount_codes.feature",
      "line": 3,
      "column": 1
    },
    "name": {
      "path": "/users/jan/demo/features/discount_codes.feature",
      "line": 9,
      "column": 3
    },
  }
}

It can be used to support both mocha and cucumber and should help the IDE do the navigation work.

The next step will be to use the TC protocol to report on given/when/then steps as well as tasks and interactions.

I think this should be a fairly clean approach. Would that work for you @segrey?

J

@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey Apr 19, 2017

Sure, that would work fine.
As for reporting the results at a scenario level rather than a step level, you're right, I missed that. However, it might be convenient to have steps in the test tree too. See a related request in WebStorm issue tracker:

I would expect to see the steps in the tree as well. I'm thinking a click would go to the step in the scenario, a shift+click or context menu would take you to the step code definition.

No idea which way is better. Good thing is that testInfo object extension will work for both approaches.

Minor: maybe sourceLocation would express more explicitly the required source attribute (e.g. it's not about source content), though source is fine. :)

segrey commented Apr 19, 2017

Sure, that would work fine.
As for reporting the results at a scenario level rather than a step level, you're right, I missed that. However, it might be convenient to have steps in the test tree too. See a related request in WebStorm issue tracker:

I would expect to see the steps in the tree as well. I'm thinking a click would go to the step in the scenario, a shift+click or context menu would take you to the step code definition.

No idea which way is better. Good thing is that testInfo object extension will work for both approaches.

Minor: maybe sourceLocation would express more explicitly the required source attribute (e.g. it's not about source content), though source is fine. :)

jan-molak added a commit that referenced this issue Apr 26, 2017

feat(core): support for ES6-style task definitions
affects: @serenity-js/cucumber-2, serenity-js

- Instead of using the @step annotation it's enough for a task to define a toString() method, which will be used to determine its description #22
- RecordedTask and RecordedActivity are now replaced by RecordedActivity, which can also define the location where it was invoked to support #18
- A more functional-style DSL provided to make prototyping faster, for example, to create a task it's enough to write:  task_where('{0} proceeds to checkout') #21

ISSUES CLOSED: #21, #22
@segrey

This comment has been minimized.

Show comment
Hide comment
@segrey

segrey May 16, 2017

@jan-molak Maybe sourceLocation.name doesn't sound great as location doesn't have name. Maybe let's just have:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
  "durationMillis": 547,
  "categorySourceLocation": {
    "path": "/users/jan/demo/features/discount_codes.feature",
    "line": 3,
    "column": 1
  },
  "nameSourceLocation": {
    "path": "/users/jan/demo/features/discount_codes.feature",
    "line": 9,
    "column": 3
  }
}

segrey commented May 16, 2017

@jan-molak Maybe sourceLocation.name doesn't sound great as location doesn't have name. Maybe let's just have:

{
  "category": "Discount codes for selected products",
  "name": "Using a discount code received from the online store",
  "durationMillis": 547,
  "categorySourceLocation": {
    "path": "/users/jan/demo/features/discount_codes.feature",
    "line": 3,
    "column": 1
  },
  "nameSourceLocation": {
    "path": "/users/jan/demo/features/discount_codes.feature",
    "line": 9,
    "column": 3
  }
}
@stavalfi

This comment has been minimized.

Show comment
Hide comment
@stavalfi

stavalfi Aug 2, 2018

guys do you have any plans for support to run a single scenario?

stavalfi commented Aug 2, 2018

guys do you have any plans for support to run a single scenario?

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Aug 8, 2018

Owner

Hey @stavalfi - both cucumber and mocha allow you to run single scenarios, as per the docs. Hope that helps!

Owner

jan-molak commented Aug 8, 2018

Hey @stavalfi - both cucumber and mocha allow you to run single scenarios, as per the docs. Hope that helps!

@stavalfi

This comment has been minimized.

Show comment
Hide comment
@stavalfi

stavalfi Aug 9, 2018

@jan-molak Thanks!

I was thinking about running specific scenario or specific feature from WebStorm and not from the command line. Sometimes the names are too long for writing them down every time.

stavalfi commented Aug 9, 2018

@jan-molak Thanks!

I was thinking about running specific scenario or specific feature from WebStorm and not from the command line. Sometimes the names are too long for writing them down every time.

@jan-molak

This comment has been minimized.

Show comment
Hide comment
@jan-molak

jan-molak Aug 10, 2018

Owner

@stavalfi - gotcha. In this case, you'll most likely need to create a run configuration. I'm not sure if IntelliJ Cucumber.js plugin supports a "right click/run scenario" feature the same way Mocha one does? @segrey?

Owner

jan-molak commented Aug 10, 2018

@stavalfi - gotcha. In this case, you'll most likely need to create a run configuration. I'm not sure if IntelliJ Cucumber.js plugin supports a "right click/run scenario" feature the same way Mocha one does? @segrey?

@stavalfi

This comment has been minimized.

Show comment
Hide comment
@stavalfi

stavalfi Aug 10, 2018

Thank you. I will try it. Also, it could be nice to running feature or scenario by writing substring and then running all the features/scenarios which contains that substring.

Thanks again @jan-molak !

stavalfi commented Aug 10, 2018

Thank you. I will try it. Also, it could be nice to running feature or scenario by writing substring and then running all the features/scenarios which contains that substring.

Thanks again @jan-molak !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment