New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make scenario TestResult public #963

Merged
merged 11 commits into from Oct 26, 2017

Conversation

Projects
None yet
4 participants
@nvborisenko
Contributor

nvborisenko commented Oct 24, 2017

We need understand the result of scenario in [AfterScenario] hook. Currently we have access only to TestError property which doesn't reflect the actual scenario result (pending or binding error).

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Oct 24, 2017

Thanks for the PR!

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Oct 24, 2017

Could you add a line to the changelog for this please?

@gasparnagy

This comment has been minimized.

Collaborator

gasparnagy commented Oct 24, 2017

I am not against it, but I would like to reconsider the naming of this enum and the property first:

  • should the enum (and the prop?) be called ScenarioExecutionStatus
  • it should be moved to the TechTalk.SpecFlow namespace (the TechTalk.SpecFlow.Infrastructure is more for internal stuff)
  • rename MissingStepDefinition to UndefinedStep to me more conform with other cukes

@SabotageAndi what do you think?

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Oct 24, 2017

I would rename the enum and its value later.
Making the property public is for me ok for a 2.2.3 release, but changing the enum is for me a breaking change that would need a 2.3 release.

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Oct 24, 2017

As we are already on course for SpecFlow 2.3, please rename the enums according @gasparnagy suggestions.

- Rename to ScenarioExecutionStatus
- Moved this to the root of the project
- UndefinedStep in enum
- Edited changelog.txt
@nvborisenko

This comment has been minimized.

Contributor

nvborisenko commented Oct 24, 2017

Done, please review, especially changelog file. Thanks in advance.

@gasparnagy

I am fine with the changes, except the small correction in the chaneglog. (We have to check why the build was failing still.)

@@ -1,3 +1,7 @@
2.3 - 2017-??-??
New Features:
+ Expose the result of scenario in ScenarioContext via ScenarioExecutionStatus property

This comment has been minimized.

@gasparnagy

gasparnagy Oct 24, 2017

Collaborator

Maybe "Expose the current status (result) of the scenario execution in ...".
The property can also be retrieved during scenario execution, so it does not necessarily show the "result" just the current status. (When retrieving it from an AfterScenario it provides the result.)

This comment has been minimized.

@nvborisenko

nvborisenko Oct 25, 2017

Contributor

Changed.

@nvborisenko

This comment has been minimized.

Contributor

nvborisenko commented Oct 25, 2017

@gasparnagy @SabotageAndi Tests are passing locally, but not on appveyor side. Based on the issue I suppose that the issue is related to concurred access to %temp% folder by several tests threads. I see your solution has 4 dlls, and you execute tests with nunit3-console runner. By default this runner uses 1 agent per dll, and maximum number of agents are 2 (because appveyor machine has 2 cpus). So I guess that some 2 parallel unit tests used %temp% folder at the same time.
I am able to reproduce similar behavior on my local environment. If I execute tests in 1 thread (agent) then all tests are green. If in several threads - I get periodical failures.
Guys, could you please change the execution configuration on appveyour? Please add --agents=1 argument for nunit3-console.exe to verify my assumption.

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Oct 25, 2017

Ah, this could be the issue of the sporadic build errors.
You can do this also. If you change the appveyor.yml in a pull request, AppVeyor uses this and not the one in the main branch.

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Oct 25, 2017

If I remember correctly, the configuration is available as environment variable. So %Configuration% could work

@nvborisenko

This comment has been minimized.

Contributor

nvborisenko commented Oct 25, 2017

And finally the build is green. Now unit tests are being executed in 1 thread.

@gasparnagy gasparnagy merged commit 8e660f9 into techtalk:master Oct 26, 2017

1 check passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
@gasparnagy

This comment has been minimized.

Collaborator

gasparnagy commented Oct 26, 2017

Autom8edChaos added a commit to Autom8edChaos/SpecFlow that referenced this pull request Nov 6, 2017

Update from original (#1)
* update MSBuild.Community.Tasks to latest version via NuGet (techtalk#801)

* Regexless stepdef fixes (techtalk#786)

* Upgrade FluentAssertions

* Add simple unit test

* Add unit test for parametrized method

* Add unit tests for regexless - 36 failing

* fluent assertion does not allow { and } in the because part

* refine expectations: shortening like "doesn't" should not work automatically

* fix regexless matching issues

* a bit more optimal solution that produces simpler regexes

* Improve param locator

* should not remove keywords from binding culture

* small code improvements

* refine expectations: should not allow whitespace or punctuation in the front of the step text (Given !foo) in order to make the generated regex faster (does not start with rule)

* small refactor

* update changelog, remove wip tags

* add test for issue techtalk#715

* Update changelog showing that techtalk#301 is also fixed

* add CI NuGet package feed

* Added support for MsTest's [DeploymentItem] attribute with @mstest:DeploymentItem tag (techtalk#805)

* Added support for MsTest's [DeploymentItem] attribute with @Deploy tag (Issue techtalk#803)
* Prefixing new tag with `MsTest:`
* using pascal case @mstest:DeploymentItem
* wrapping nested using blocks with curly brackets

* Change project reader from MSBuild to XML (techtalk#837)

* improve tests for project reader
* add tests for new csproj format
* change variable naming
* complete read of classic projects
* refactoring
* mark not used properties as obsolete
* support new project system & refactorings
* use AppDomainIsolatedTask instead of Task to solve file looking issues with MsBuild
* fix typo :-/
* add tests for linked files

* Added display name for theory attribution (techtalk#812)

* Added display name for theory attribution
* Fixed incorrect namespace reference to TheoryAttribute (no longer part of extensions)

* Support context parameters on Before/After methods, FeatureContainer (techtalk#779)

* Added scenario: Inject FeatureContext into a BeforeFeature hook
* first dummy implementation to make the scenario pass (runtime tests failing)
* Use bindingInstanceResolver to resolve parameters
* make unit tests pass
* Add unit tests for what we have so far
* support for resolving hook parameters from scenario context
* better then step for the scenarios
* fix unit test error
* added failing test for resolving objects from feature container
* refactor ScenarioContext
* Implement FeatureContainer
* move binding culture initialization to FeatureContext
* fix featurecontext resolution from scenario container
* Refactor scenario and feature context and remove displose hack that was necessary to avoid circular disposing loops
* fix unit test failure
* Make InternalContextManager to IObjectContainer reference more explicit
* rename IBindingInstanceResolver to ITestObjectResolver (breaking change for some plugins)

* Wildcard support for project reader (techtalk#843)

* Wildcard support for project reader
* fix tests for AppVeyor

* update changelog

* Config file refactoring & Json Config file support (techtalk#690)

* split app.config elements to multiple files

* make RuntimeConfiguration nearly immutable

* extract config loading logic to RuntimeConfigurationLoader

* reformating

* add tests for app.config reading

* move ConfigurationTest to separate Namespace

* rename file

* add json tests

* add json.net dependency

* json config file parsing

* add tests for checking which config file is used

* refactoring for tests and added tests

* remove GeneratorConfiguration

* fix dependencies for generator

* fix tests

* fix tests

* move files & fix namespaces

* work on generation config loading

* fix tests and make parsing more robust

* fix and comment test

* small cleanups

* remove ToolLanguage, GeneratorPath and DetectAmbiguousMatches from Config

rename PrintConfigSource to TraceConfigSource

* fix merge issues

* remove file

* add missing json handling

* update changelog

* add Newtonsoft.Json dependency to the NuGet package

* update TestGenerator version

* fix typo in README.md (techtalk#853)

* fix VS code behind generation (techtalk#855)

* fix redirects
* fix serialization problem with Visual Studio
* fix lineending

* Order sensitive overload for compare to set (techtalk#778)

* Add tests
* Refactor TableDifferenceResults to be able to represent order sensitive diff
* make throw tests pass
* refactor message tests to be able to test order sensitive comparison
* fix diff messages for order sensitive
* Replace Tuple with TableDifferenceItem

* Hook with multiple Tags scoped are executed more than once (techtalk#848)

* add failing tests
* Fix for techtalk#848 Hook with multiple Tags scoped are executed more than once

* making sure that the ordering is preserved (the GroupBy might change the order)

* update changelog for 2.2.0-prerelease20170523

* add GitExtensions configuration file with github and appveyor configuration (techtalk#862)

* Added TestThreadContext and exposed it in ContextManager and Steps base class (techtalk#875)

* Added TestThreadContext and exposed it in ContextManager and Steps base class

* Upgrade to BoDi v1.3 (techtalk#876)

* fix shadow copy test issue
* upgrade BoDi to v1.3
* remove unnecessary BoDi workarounds

* Adds support for xUnit2 ITestOutputHelper techtalk#575 (techtalk#874)

* Resolves techtalk#575 

* added tests for adding support for xUnit2 ITestOutputHelper class

added XUnit2Generator unit tests
added XUnit2Provider specs
updated XUnitExecutionDriver to output results in default (xUnit) xml
output instead of NUnit xml format

* swapped the order of the TestInitialize call and the setting of the _testOutputHelper field

* expose TestAssembly and BindingAssemblies on ITestRunnerManager

* Fix generation from visual studio (techtalk#877)

* rename field so that it matches the version of this class from 1.9 (which is used by VS)
* make AppConfig the default value
* add warnings

* Update for 2.2 release

* use FileStream constructor which grants ReadWrite to other processes. (techtalk#906)

use FileStream constructor which grants ReadWrite to other processes

techtalk#904

* Added changelog entry for PR (techtalk#907)

* correct version number

* Access MSTest's TestContext property via ScenarioContext (techtalk#882)

* injected MSTest's TestContext into ScenarioContext

* fixed TestContext property generation for VB.Net

* @MSBuild [DeploymentItem] - added option to provide output directory (techtalk#901)

* @MSBuild [DeploymentItem] - added option to provide output directory

* code review

* update changelog after merging PRs techtalk#882 & techtalk#901

* add tests for Feature/Scenario Hooks with context parameters (techtalk#925)

* Issue Template (techtalk#924)

* Create issue template

* add SpecFlow+Runner to test runners

* Rename issue to ISSUE_TEMPLATE.md

* Allowed custom XSLT scripts to contain scripts.

* Updated changelog for new addition

* enter release date

* fix changelog after 2.2.1 release

* added link to Stack Overflow repro topic (techtalk#942)

* Avoid trying to load empty configuration

* Set ConfigSource of default holder to "Default" instead of "AppConfig"

* Turn off line ending git auto conversion (techtalk#953)

* Stop git from automatically converting line endings everytime repo is cloned
* add .vs folder to .gitignore

* Fixing copy link in step definition report (techtalk#958)

* Update changelog

* Support for tuples (techtalk#951)

* Initial support for Tuples
* Adding nuspec dependency
* Added tupple error when there are too many properties

* add entry for tuple

* Make scenario TestResult public (techtalk#963)

* Make scenario TestResult public

* - Rename to ScenarioExecutionStatus
- Moved this to the root of the project
- UndefinedStep in enum
- Edited changelog.txt

* Update changelog.txt

* Single agent for unit tests execution

* Fix incorrect appveyour config

* Try to change test_script to test

* Disable automatic tests

* Provide full path to test assemblies

* Expose configuration var to batch

* Hardcoded configuration as Release

* Use %% accessor for configuration var

* Add hyperlink to discussion group (techtalk#965)

Remove ambiguity regarding where to go for questions.
@yurii-hunter

This comment has been minimized.

yurii-hunter commented Nov 9, 2017

@SabotageAndi, @gasparnagy Do we have a roadmap for releases? In particular when are you planning to have 2.3 version?

@SabotageAndi

This comment has been minimized.

Collaborator

SabotageAndi commented Nov 9, 2017

There is no roadmap.
I am looking forward to do a 2.3 release this year.

@SabotageAndi SabotageAndi added this to the SpecFlow 2.3 milestone Dec 12, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment