Cucumber4: Obtain Cucumber 3 scenario properties in hooks.rb

[orien]

Describe the bug

In version 3 it is possible to obtain the name of the current feature from Before and After hooks.

Before do |scenario|
  feature_name = scenario.feature.name
  #…
end

In version 4 this seems impossible as the feature method has been removed and I can see no replacement.

Before do |scenario|
  feature_name = scenario.feature.name
  # -> undefined method `feature' for #<Cucumber::RunningTestCase::TestCase:0x00007f9b307ae180> (NoMethodError)
end

To Reproduce
Steps to reproduce the behavior:

1. Install version 4
2. Add Before hook obtaining feature name
3. Run cucumber
4. See error

Expected behavior

Be able to obtain the running feature name from Before and After hooks.

Context & Motivation

Providing a replacement for this would ease the migration path to version 4.0.0 as integrations rely on this data.

For example, the VCR gem uses the scenario and feature names to determine which cassette to load.

[luke-hill]

Do other methods on the scenario object work. As it seems to be much different.

Examples are scenario.source_tag_names scenario.name e.t.c.?

[orien]

I see the following methods:

Cucumber 3

Before do |s|
  s.exception # => #<NilClass>
  s.feature # => #<Cucumber::Core::Ast::Feature>
  s.keyword # => #<String>
  s.language # => #<Gherkin::Dialect>
  s.location # => #<struct Cucumber::Core::Ast::Location::Precise
  s.name # => #<String>
  s.source # => [#<Cucumber::Core::Ast::Feature>, #<Cucumber::Core::Ast::Scenario>]
  s.source_tag_names # => [#<String>]
  s.status # => #<Symbol>
  s.step_count # => #<Integer>
  s.tags # => [#<Cucumber::Core::Ast::Tag>]
  s.test_steps # => [#<Cucumber::Core::Test::Step>, #<Cucumber::Core::Test::Step>]
end

Cucumber 4

Before do |s|
  s.exception # => #<NilClass>
  s.hash # => #<Integer>
  s.id # => #<String>
  s.language # => #<String>
  s.location # => #<struct Cucumber::Core::Test::Location::Precise
  s.name # => #<String>
  s.source_tag_names # => [#<String>]
  s.status # => #<Symbol>
  s.step_count # => #<Integer>
  s.tags # =>[#<Cucumber::Core::Test::Tag>]
  s.test_steps # => [#<Cucumber::Core::Test::Step>, #<Cucumber::Core::Test::HookStep>]
end

[luke-hill]

Very comprehensive answer - thankyou. I'd definitely expect we could keep the old methods, but I've not been involved much with it recently.

[JoeSSS]

Are there any workarounds to get feature name inside the scenario? We really want to update to new Cucumber version but feature name is essential for our tracking and flakiness monitoring on test results BE.

[thedeeno]

If anyone wants a work-around for VCR while this is being considered I hacked up the following monkey-patch:

# HACK: this method was available in cucumber 3.1 and VCR relies on it to
# generate cassette names.
module Cucumber
  module RunningTestCase
    class TestCase < SimpleDelegator

      def feature
        string = File.read(location.file)
        document = ::Gherkin::Parser.new.parse(string)
        document[:feature]
      end

    end
  end
end

There's probably a better way to do this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[atyndall]

I believe this is still an issue?

[luke-hill]

It is yes, but it is distinctly non-trivial and would require an agreement from the technical team on "what" we should do and whether we standardise across the flavours. I'll pop a slow-burner on it.

[Castone22]

I was doing some quick digging into the JSON formatter because SOMEHOW it still gets the feature name, seems like the expected way to handle it now is to use an AstLookup...

cucumber-ruby/lib/cucumber/formatter/json.rb

Line 26 in e318dfc

@ast_lookup = AstLookup.new(config)

which gets referenced here

cucumber-ruby/lib/cucumber/formatter/json.rb

Line 38 in e318dfc

builder = Builder.new(test_case, @ast_lookup)

The builder uses this to create a feature hash which can be cross referenced with the test location to get its feature name.

cucumber-ruby/lib/cucumber/formatter/json.rb

Lines 240 to 247 in e318dfc

	def initialize(test_case, ast_lookup)
	@background_hash = nil
	uri = test_case.location.file
	feature = ast_lookup.gherkin_document(uri).feature
	feature(feature, uri)
	background(feature.children.first.background) unless feature.children.first.background.nil?
	scenario(ast_lookup.scenario_source(test_case), test_case)
	end

Should be able to intercept the config in an AfterConfig hook and save it for use i'd think.

[luke-hill]

@Castone22 That is not using the new message protocol. So it's not a like for like comparison. The legacy JSON formatter is being removed (It was slated for v5 removal, I believe this is now pushed back to v6)

[Castone22]

I suppose that explains why the ast from the ast lookup consists of protofbufs, do you happen to know retrieving those is also slated to be removed?

[rickpastoor]

It appears that the patch by @thedeeno no longer works with cucumber 7.0.0. Investigating further, but if anyone already found the solution I'd love to hear it :)

[luke-hill]

The internal structure of cucumber messages has basically been rewritten from scratch in cucumber 7. We now use a JSON schema instead of protobufs. Consequently, it's likely a lot of the internal structure has changed.

[rickpastoor]

This patch works for me:

module Cucumber
  module RunningTestCase
    class TestCase < SimpleDelegator

      def feature
        string = File.read(location.file)
        document = ::Gherkin::Parser.new.parse(string)
        document.feature
      end

    end
  end
end

[aurelien-reeves]

I put both "Bug" and "Enhancement" labels because historically, it is a regression when cucumber v4 has been released, but regarding all the internal rework which led to the actual v7, internally it would actually be an enhancement 😅

[luke-hill]

You can still access the scenario name, the scenario location (Which includes a fully qualified filepath), and things like tags.

The switchover in mentality (I was against it at first too), is that the pickles execute the test cases, i.e. they should be thought of as the top level wrapper (or a scenario if thats easier). The feature isn't a realistic wrapper at this point.

I mentioned VCR were one of the larger consumers of this, I think maybe using a scenario name to qualify your VCR cassettes might help.

[Castone22]

Another use case is having a custom formatter that logs metadata such as relative feature location, id, name and scenario gherkin plus the pass and fail state to with error output and such to an elk instance. (Something we're planning to open source in the near future.)

I'd expect other report tooling that hooks into things like zephyr scale or x-ray would benefit from a way to access this information. We're often more interested in the feature ast data in these situations though. I don't necessarily care about the form the data takes, Json is completely acceptable. We used to be able to resolve this through scenario.feature.source

[mattwynne]

@Castone22 the formatter / events API is a better fit for that, though it's probably not very well documented at the moment. There are examples in Cucumber's own codebase were we match together the data from the test case result and Gherkin AST events to give the kind of info you're looking for, e.g. https://github.com/cucumber/cucumber-ruby/blob/97a03c0bd8199d5dcd3fc8de5993d01306dd4a1f/lib/cucumber/formatter/pretty.rb

It would be possible to build a richer domain model from these events to surface into a user API, but it's not something we have at the moment.

[Castone22]

Yes those is what I ultimately ended up using. A lack of a direct link from a test case to it's ast source was a little awkward at first though using an ast lookup was the pattern I was referring to, not the protobuf that resulted from using it back in 5. I was waiting for confirmation this was the intended route before I continued implementing against it though.

[mattwynne]

@brasmusson really clarified it for me today at the Zoom community meeting, that the hooks are part of an execution model, and we should keep the info exposed in them lean because, for example, we might have test cases that were generated from sources other than Gherkin documents, or test cases running in parallel.

On the other hand, the events API used by the formatters is the higher-level place where you can observe everything going on in the Cucumber run, including AST nodes and execution results.

I think we're probably missing an abstraction layer that makes it easier to work with the events API for common use cases, so please keep talking to us and see if there are patterns in what you do with it that could be pushed upstream.

[aurelien-reeves]

Thanks for all those explanations everyone!

What is that VCR project you are talking about?

[luke-hill]

@aurelien-reeves https://github.com/vcr/vcr it's quite a large gem used in lots of projects (Like rails testing for example).

[aurelien-reeves]

Actually we still have to rename the paramter from "scenario" to "test_case" to reduce the confusion

[aurelien-reeves]

I took a look in order to rename the parameter from "scenario" into "test_case" for hooks.
I was not able to find in the code any relevant place.

@luke-hill do you know if it is only matter of documentation? Or does the code need some update too?

Reminder: in order to be more clear than the hook parameter represent a pickle and not a scenario, we wanted to rename Before do |scenario| into Before do |test_case|

[luke-hill]

in code it's kinda irrelevant as it's just a placeholder name.

Just in ruby these names invoke meaning. So if the param is called scenario it would be reasonable to assume you could get scenario based properties from them.

This is purely a doc issue now.

[aurelien-reeves]

in code it's kinda irrelevant as it's just a placeholder name.

Just in ruby these names invoke meaning. So if the param is called scenario it would be reasonable to assume you could get scenario based properties from them.

That's the point: I did not find any code in the repo which had a param named 'scenario'. So I guess our own codebase is ok

This is purely a doc issue now.

👍

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two months if no further activity occurs.

[luke-hill]

I believe we fixed the docs part of this so this needs closing again.

Your code above is slightly faulty @RamadhanaRey - You want to run respond_to for both. But in essence, we're not going back to the old system. So just use the scenario name.

If you need the feature file name: You can obtain it here: https://github.com/site-prism/automation_helpers/blob/main/lib/automation_helpers/extensions/cucumber/core/test/case.rb#L13-L15

[abhiras29]

went through the whole thread and I know this is marked as completed
but could not find any solution for scenario.outline?
I know this method got deprecated in the new version of cucumber but any hack like we have for feature.name
help much appreciated

[Castone22]

I know you can figure it out using an ast lookup like the one I showed above, outlines use a different class then regular scenarios