Make scenario TestResult public #963
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for the PR! |
|
Could you add a line to the changelog for this please? |
|
I am not against it, but I would like to reconsider the naming of this enum and the property first:
@SabotageAndi what do you think? |
|
I would rename the enum and its value later. |
|
As we are already on course for SpecFlow 2.3, please rename the enums according @gasparnagy suggestions. |
- Moved this to the root of the project - UndefinedStep in enum - Edited changelog.txt
|
Done, please review, especially changelog file. Thanks in advance. |
SabotageAndi
approved these changes
Oct 24, 2017
gasparnagy
requested changes
Oct 24, 2017
changelog.txt
Outdated
| @@ -1,3 +1,7 @@ | |||
| 2.3 - 2017-??-?? | |||
| New Features: | |||
| + Expose the result of scenario in ScenarioContext via ScenarioExecutionStatus property | |||
There was a problem hiding this comment.
Maybe "Expose the current status (result) of the scenario execution in ...".
The property can also be retrieved during scenario execution, so it does not necessarily show the "result" just the current status. (When retrieving it from an AfterScenario it provides the result.)
|
@gasparnagy @SabotageAndi Tests are passing locally, but not on appveyor side. Based on the issue I suppose that the issue is related to concurred access to %temp% folder by several tests threads. I see your solution has 4 dlls, and you execute tests with nunit3-console runner. By default this runner uses 1 agent per dll, and maximum number of agents are 2 (because appveyor machine has 2 cpus). So I guess that some 2 parallel unit tests used %temp% folder at the same time. |
|
Ah, this could be the issue of the sporadic build errors. |
|
If I remember correctly, the configuration is available as environment variable. So %Configuration% could work |
|
And finally the build is green. Now unit tests are being executed in 1 thread. |
gasparnagy
approved these changes
Oct 26, 2017
|
thx @nvborisenko! |
Autom8edChaos
added a commit
to Autom8edChaos/SpecFlow
that referenced
this pull request
Nov 6, 2017
* update MSBuild.Community.Tasks to latest version via NuGet (SpecFlowOSS#801) * Regexless stepdef fixes (SpecFlowOSS#786) * Upgrade FluentAssertions * Add simple unit test * Add unit test for parametrized method * Add unit tests for regexless - 36 failing * fluent assertion does not allow { and } in the because part * refine expectations: shortening like "doesn't" should not work automatically * fix regexless matching issues * a bit more optimal solution that produces simpler regexes * Improve param locator * should not remove keywords from binding culture * small code improvements * refine expectations: should not allow whitespace or punctuation in the front of the step text (Given !foo) in order to make the generated regex faster (does not start with rule) * small refactor * update changelog, remove wip tags * add test for issue SpecFlowOSS#715 * Update changelog showing that SpecFlowOSS#301 is also fixed * add CI NuGet package feed * Added support for MsTest's [DeploymentItem] attribute with @mstest:DeploymentItem tag (SpecFlowOSS#805) * Added support for MsTest's [DeploymentItem] attribute with @Deploy tag (Issue SpecFlowOSS#803) * Prefixing new tag with `MsTest:` * using pascal case @mstest:DeploymentItem * wrapping nested using blocks with curly brackets * Change project reader from MSBuild to XML (SpecFlowOSS#837) * improve tests for project reader * add tests for new csproj format * change variable naming * complete read of classic projects * refactoring * mark not used properties as obsolete * support new project system & refactorings * use AppDomainIsolatedTask instead of Task to solve file looking issues with MsBuild * fix typo :-/ * add tests for linked files * Added display name for theory attribution (SpecFlowOSS#812) * Added display name for theory attribution * Fixed incorrect namespace reference to TheoryAttribute (no longer part of extensions) * Support context parameters on Before/After methods, FeatureContainer (SpecFlowOSS#779) * Added scenario: Inject FeatureContext into a BeforeFeature hook * first dummy implementation to make the scenario pass (runtime tests failing) * Use bindingInstanceResolver to resolve parameters * make unit tests pass * Add unit tests for what we have so far * support for resolving hook parameters from scenario context * better then step for the scenarios * fix unit test error * added failing test for resolving objects from feature container * refactor ScenarioContext * Implement FeatureContainer * move binding culture initialization to FeatureContext * fix featurecontext resolution from scenario container * Refactor scenario and feature context and remove displose hack that was necessary to avoid circular disposing loops * fix unit test failure * Make InternalContextManager to IObjectContainer reference more explicit * rename IBindingInstanceResolver to ITestObjectResolver (breaking change for some plugins) * Wildcard support for project reader (SpecFlowOSS#843) * Wildcard support for project reader * fix tests for AppVeyor * update changelog * Config file refactoring & Json Config file support (SpecFlowOSS#690) * split app.config elements to multiple files * make RuntimeConfiguration nearly immutable * extract config loading logic to RuntimeConfigurationLoader * reformating * add tests for app.config reading * move ConfigurationTest to separate Namespace * rename file * add json tests * add json.net dependency * json config file parsing * add tests for checking which config file is used * refactoring for tests and added tests * remove GeneratorConfiguration * fix dependencies for generator * fix tests * fix tests * move files & fix namespaces * work on generation config loading * fix tests and make parsing more robust * fix and comment test * small cleanups * remove ToolLanguage, GeneratorPath and DetectAmbiguousMatches from Config rename PrintConfigSource to TraceConfigSource * fix merge issues * remove file * add missing json handling * update changelog * add Newtonsoft.Json dependency to the NuGet package * update TestGenerator version * fix typo in README.md (SpecFlowOSS#853) * fix VS code behind generation (SpecFlowOSS#855) * fix redirects * fix serialization problem with Visual Studio * fix lineending * Order sensitive overload for compare to set (SpecFlowOSS#778) * Add tests * Refactor TableDifferenceResults to be able to represent order sensitive diff * make throw tests pass * refactor message tests to be able to test order sensitive comparison * fix diff messages for order sensitive * Replace Tuple with TableDifferenceItem * Hook with multiple Tags scoped are executed more than once (SpecFlowOSS#848) * add failing tests * Fix for SpecFlowOSS#848 Hook with multiple Tags scoped are executed more than once * making sure that the ordering is preserved (the GroupBy might change the order) * update changelog for 2.2.0-prerelease20170523 * add GitExtensions configuration file with github and appveyor configuration (SpecFlowOSS#862) * Added TestThreadContext and exposed it in ContextManager and Steps base class (SpecFlowOSS#875) * Added TestThreadContext and exposed it in ContextManager and Steps base class * Upgrade to BoDi v1.3 (SpecFlowOSS#876) * fix shadow copy test issue * upgrade BoDi to v1.3 * remove unnecessary BoDi workarounds * Adds support for xUnit2 ITestOutputHelper SpecFlowOSS#575 (SpecFlowOSS#874) * Resolves SpecFlowOSS#575 * added tests for adding support for xUnit2 ITestOutputHelper class added XUnit2Generator unit tests added XUnit2Provider specs updated XUnitExecutionDriver to output results in default (xUnit) xml output instead of NUnit xml format * swapped the order of the TestInitialize call and the setting of the _testOutputHelper field * expose TestAssembly and BindingAssemblies on ITestRunnerManager * Fix generation from visual studio (SpecFlowOSS#877) * rename field so that it matches the version of this class from 1.9 (which is used by VS) * make AppConfig the default value * add warnings * Update for 2.2 release * use FileStream constructor which grants ReadWrite to other processes. (SpecFlowOSS#906) use FileStream constructor which grants ReadWrite to other processes SpecFlowOSS#904 * Added changelog entry for PR (SpecFlowOSS#907) * correct version number * Access MSTest's TestContext property via ScenarioContext (SpecFlowOSS#882) * injected MSTest's TestContext into ScenarioContext * fixed TestContext property generation for VB.Net * @MSBuild [DeploymentItem] - added option to provide output directory (SpecFlowOSS#901) * @MSBuild [DeploymentItem] - added option to provide output directory * code review * update changelog after merging PRs SpecFlowOSS#882 & SpecFlowOSS#901 * add tests for Feature/Scenario Hooks with context parameters (SpecFlowOSS#925) * Issue Template (SpecFlowOSS#924) * Create issue template * add SpecFlow+Runner to test runners * Rename issue to ISSUE_TEMPLATE.md * Allowed custom XSLT scripts to contain scripts. * Updated changelog for new addition * enter release date * fix changelog after 2.2.1 release * added link to Stack Overflow repro topic (SpecFlowOSS#942) * Avoid trying to load empty configuration * Set ConfigSource of default holder to "Default" instead of "AppConfig" * Turn off line ending git auto conversion (SpecFlowOSS#953) * Stop git from automatically converting line endings everytime repo is cloned * add .vs folder to .gitignore * Fixing copy link in step definition report (SpecFlowOSS#958) * Update changelog * Support for tuples (SpecFlowOSS#951) * Initial support for Tuples * Adding nuspec dependency * Added tupple error when there are too many properties * add entry for tuple * Make scenario TestResult public (SpecFlowOSS#963) * Make scenario TestResult public * - Rename to ScenarioExecutionStatus - Moved this to the root of the project - UndefinedStep in enum - Edited changelog.txt * Update changelog.txt * Single agent for unit tests execution * Fix incorrect appveyour config * Try to change test_script to test * Disable automatic tests * Provide full path to test assemblies * Expose configuration var to batch * Hardcoded configuration as Release * Use %% accessor for configuration var * Add hyperlink to discussion group (SpecFlowOSS#965) Remove ambiguity regarding where to go for questions.
|
@SabotageAndi, @gasparnagy Do we have a roadmap for releases? In particular when are you planning to have 2.3 version? |
|
There is no roadmap. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We need understand the result of scenario in [AfterScenario] hook. Currently we have access only to TestError property which doesn't reflect the actual scenario result (pending or binding error).