🐛 Bug: Contexts not properly reset between tests within the same level

[moll]

Hey,

I wouldn't be surprised if related to my years old bug report and gripe: #1195

var assert = require("assert")

describe("Parent", function() {
  beforeEach(function() { this.value = 42 })

  it("must pass 1", function() { assert.equal(this.value, 42) })
  it("must pass 2", function() { assert.equal(this.value, 42) })

  describe("Child", function() {
    beforeEach(function() {
      assert.equal(this.value, 42)
      this.value = 13
    })

    it("must pass 1", function() { assert.equal(this.value, 13) })
    it("must pass 2", function() { assert.equal(this.value, 13) })
  })
})

Fails with:

  1) Parent Child "before each" hook for "must pass 2":

      AssertionError: 13 == 42
      + expected - actual

      -13
      +42

      at Context.<anonymous> (test/index_test.js:11:14)

[danielstjules]

What are you expecting in terms of behavior? With a simpler example, would you expect this to pass:

var assert = require('assert');

describe('Suite', function() {
  it('test1', function() {
    this.value = 'foo';
    assert.equal(this.value, 'foo');
  });

  it('test2', function() {
    assert.notEqual(this.value, 'foo');
  });
});

  Suite
    ✓ test1
    1) test2


  1 passing (12ms)
  1 failing

  1) Suite test2:

      AssertionError: 'foo' != 'foo'
      + expected - actual

It's always been my understanding that the properties are shared across tests.

[moll]

I would expect to not have this.value set in "test2". Sharing properties between tests such a way is despicable. Even if they were left there, assignments in beforeEach must take precedence. ;)

[danielstjules]

I agree that sharing state between tests is a bad idea :) But it's up to you to clean up state in an afterEach/beforeEach hook. I see these two suites as equal:

var assert = require('assert');

describe('Suite1', function() {
  it('test1', function() {
    this.value = 'foo';
    assert.equal(this.value, 'foo');
  });

  it('test2', function() {
    assert.notEqual(this.value, 'foo');
  });
});

describe('Suite2', function() {
  var value;

  it('test1', function() {
    value = 'foo';
    assert.equal(value, 'foo');
  });

  it('test2', function() {
    assert.notEqual(value, 'foo');
  });
});

  Suite1
    ✓ test1
    1) test2

  Suite2
    ✓ test1
    2) test2

There's nothing too fancy going on here.

Even if they were left there, assignments in beforeEach must take precedence. ;)

Not sure what you're talking about

[moll]

Thanks for the quick response.

But it's up to you to clean up state in an afterEach/beforeEach hook.

That doesn't make sense. Contexts need to be reset automatically. That's their whole point: to save from a mind numbing number of afterEach(function() { delete this.a; delete this.b }). Don't you agree the current behavior is just weird: beforeEaches clearly run before each test, but their assignments don't show up, as is visible in my first post?

[moll]

Context behavior primary and edge cases have been solved half, if not a full, decade ago in other languages and their libraries. ;-) No need for us to solve the wheel, wrong. The same example from RSpec, passing:

describe "Parent" do
  before(:each) do @value = 42 end

  it "must pass 1" do @value.should == 42 end
  it "must pass 2" do @value.should == 42 end

  describe "Child" do
    before(:each) do
      @value.should == 42
      @value = 13
    end

    it "must pass 1" do @value.should == 13 end
    it "must pass 2" do @value.should == 13 end
  end
end

% rspec -f spec context_spec.rb

Parent
  must pass 1
  must pass 2
  Child
    must pass 1
    must pass 2

Finished in 0.00141 seconds
4 examples, 0 failures

[danielstjules]

Ah, my mistake, misunderstood the first example. :) PR would be welcome, but it would need to target a major release, since it's a breaking change if done correctly. Might be easier to solve with the mocha 3/4 rewrite.

[danielstjules]

Ideally, this solution would have been implemented, rather than merging #1164 As mentioned, any changes to this behavior will be a potentially breaking change. But the current behavior is broken enough that it might just be a bug fix + patch release.

[moll]

Good. Looks as if that solution is equivalent to what I proposed almost two years ago.

[danielstjules]

And the solution I linked to was proposed in 2012 - no need for snide remarks :) I'm in agreement.

[moll]

Ah, indeed. Aimed to be totally neutral. Merely didn't bother with emoticons. :-)

[jbnicolai]

👍 totally agree with the issue, and that this behaviour should be changed.

[boneskull]

dunno about all of this. if we are going to break the API, I'd like to break it in such a way that functions do not run in a context.

[boneskull]

I would strongly urge users to not use this for storing values, ever.

[moll]

Until there's a functional (as opposed to imperative) alternative, it beats reassigning to variables from outer scopes. For one, it prevents accidentally reusing without re-initialization, the thing this bug is about.

[boneskull]

@moll I disagree. I always write all of my tests using function scopes if I need to make a value available. Indeed, this is pretty foolproof:

describe('something', () => {
  let foo;
  beforeEach(() => {
    foo = {};
  });

  it('should do something with foo...', () => {
    ///
  });
});

I've never understood the necessity of using this to make a value available. It leads to problems, which is why we have bugs like this one.

[boneskull]

related: in some other ticket (forget which) we're talking about the future of the BDD API and whether these callbacks should run in a context (I don't think they should).

a potential plan would be

• BDD UI is its own package wherein the current behavior is supported (v1)
• Mocha v3 releases with BDD UI v1
• BDD UI bumps major to use lambdas (v2)
• Users can upgrade to BDD UI v2 if they wish
• Mocha v4 releases with BDD UI v2

[boneskull]

(users should still be able to upgrade to latest mocha while keeping their BDD API version held back)

[boneskull]

I don't want to start a holy war, but I've never seen a suite which stores values in this that cannot be represented equivalently using function scope. If such a thing exists, I'd like to know about it ...

[moll]

The current (design) bug isn't because contexts are implemented; it's because they're implemented poorly. ;)

I would argue contexts provide stronger isolation. Should you have an async bug in a test, that foo might've been overwritten by the time something in a test uses it. Awful to debug. Also, contexts provide an easier path to parallel tests. If the whole block within a describe were to be run again for each parallel run, that wouldn't be an issue.

Contexts are useful for both shared before-eaches and shared tests. E.g. for the former:

beforeEach(require("./_signin"))

it("must respond with 200", function*() {
  var res = yield request("/", {user: this.user}))
  res.statusCode.must.equal(200)
})

Saves writing boilerplate to extract this.user to lexical vars.

[moll]

Fundamentally, yes, thises and lexicals are interchangable. I prefer the former for succinctness, but fortunately they're not mutually exclusive. There's little reason to not keep them.

[boneskull]

I don't see above where this is necessary. You don't have to use this. You can use any object and put it wherever you want.

[moll]

Imagine the file _signin.js doing this:

module.export = function*() {
  this.user = yield Users.create()
  this.session = yield Sessions.create({user: this.user})
}

In that light, my comment of boilerplate might make more sense:

Saves writing boilerplate to extract this.user to lexical vars.

[boneskull]

@moll I'll concede you can save two or three lines by using this. I'm not convinced it's worth the bother; we'll have to agree to disagree.

There's little reason to not keep them.

There is, and that's support for lambdas. You should be able to use them in your tests and still be able to control things like how long the test takes to timeout. You should also be free to bind your functions to whatever context you please.

So, this is why I want to break the API (eventually). Functions would run w/o contexts, but the first parameter is a test object. One could (ab)use the test object however they please.

example:

describe('foo', () => {
  it('should complete after awhile', (test, done) => {
    test.timeout(10000);
    setTimeout(done, 9000);
  }); 
});

But I think I'm way out on a tangent. I think the behavior you have outlined in the original issue is incorrect and should be fixed. Regardless of whether you're using test or this, sibling tests should not share context.

[drazisil]

Has a merged PR and no further activity in a year, closing.

[danielstjules]

@drazisil that PR was merged into yeoman.github.io, not mocha. The issue should probably be left open? :)

[drazisil]

@danielstjules Ah, good catch. Sorry about that and yes.

[stale]

I am a bot that watches issues for inactivity.
This issue hasn't had any recent activity, and I'm labeling it stale. In 14 days, if there are no further comments or activity, I will close this issue.
Thanks for contributing to Mocha!

[moll]

I'm sure still am interested in this.

[brianedelman]

+1 still interested

[ORESoftware]

I looked at the original bug, here is the bug, I just changed the names of the test cases to make it more clear:

var assert = require('assert');

describe('Parent', function () {

  beforeEach(function () {
    this.value = 42
  });

  it('must pass 1', function () {
    assert.equal(this.value, 42)
  });

  it('must pass 2', function () {
    assert.equal(this.value, 42)
  });

  describe('Child', function () {

    beforeEach(function () {
      assert.equal(this.value, 42);
      this.value = 13
    });

    it('must pass 3', function () {
      assert.equal(this.value, 13)
    });

    it('must pass 4', function () {
      assert.equal(this.value, 13)    // <<<< this assertion throws
    });

  });

});

pretty weird bug, don't know how that could be caused

[ScottFreeCode]

I think what's happening here is something like:

• The outer suite has an object used for this.
• The inner suite has a separate object used for this.
• The inner suite's this object is prototype-chained to the outer suite's this object.
• So when the inner suite reads properties of this before having set them, it sees the properties set by the outer suite.
• But when it sets properties, it sets them on its own object and not the outer suite's object.
• The outer suite then doesn't overwrite the inner suite's object's properties.
• And finally, the inner suite reads the property again but this time it's only seeing its own object's property and not the outer suite's object's property.

This is, obviously, rather different from how RSpec was implemented; I haven't gotten around to double-checking how other RSpec-like JavaScript test runners (especially those that claim compatibility with Mocha) treat this.

---

Recently I was thinking about how Mocha's settings are designed and I wonder if these things might be related. Mocha has a lot of settings like retries and timeout that work like so:

• They have a global setting from the CLI or the programmatic API.
• Then each suite may override that setting.
• Nested/inner suites may override their containing/outer suite's settings.
• Other members of a suite (tests, other nested suites) don't see the overrides in any nested suites other than themselves.
• (And finally, in many cases individual tests/hooks can override the setting just for themselves.)

I'm not sure whether the last point fits, but I wonder if the settings overrides are implemented through a prototype chain too -- and if so, whether these two mechanisms just happen to be similar (perhaps not coincidentally inasmuch as they were both products of the same design pattern) or whether they're actually technically related in the code (which could make it a lot harder to change even if we want to, since we'd have to implement some kind of decoupling between this and the settings overrides).

(I haven't actually learned how some of this stuff is implemented really deep under the hood, despite working with Mocha for a couple years now. The bowels of the code are sometimes hard to figure out.)

[moll]

The outer suite then doesn't overwrite the inner suite's object's properties.

Just to remind people, I think I covered the proper implementation of this 3 years ago in #1195. :)

[ScottFreeCode]

While I'm in this particular conversation...

I've never seen a suite which stores values in this that cannot be represented equivalently using function scope. If such a thing exists, I'd like to know about it ...

You won't find any such thing in Mocha since the suite-level prototype chain ensures that this behaves identically to suite-level vars (unless you read the outer suite's value before setting the inner suite's, which isn't possible with var as far as I know); any real advantage this might have over var requires changes to the behavior of this (e.g. isolating the properties set from one test to the next, or allowing outer Each hooks to see values overridden in inner-suite non-Each hooks so that Each behavior is defined once in the outer suite but customized in the inner suites).

[c1moore]

Just to remind people, I think I covered the proper implementation of this 3 years ago in #1195. :)

I disagree. Please see #3109 I opened a few weeks ago. Prototypical this really doesn't solve the problem of context remaining between tests.

Allowing outer Each hooks to see values overridden in inner-suite non-Each hooks so that Each behavior is defined once in the outer suite but customized in the inner suites

If something is defined in a non-Each hook, it should not be customized in inner suites. That defeats the whole purpose of removing state (context) across tests. Non-Each hooks should set up the environment: connecting to the DB, starting the server, requiring external modules, etc. If you want to define a variable (even a class) for your tests, that should happen in Each blocks, so as to ensure state (context) is not preserved. Sure it may take a few extra milliseconds, but the purpose of tests isn't really to write performant code, but test (hopefully) accurate code.

If the concern is for those that have lost their way off the path of righteousness, I say we help lead them back to the light with a major version bump and that's it. If a medieval cleansing is too cruel, perhaps we even add some documentation explaining why maintaining context is bad . The golden handcuffs of compatibility will either make fixing this inefficient or not really a fix (or both). However, if backwards-compatibility is really a necessity, perhaps we can try:

• After each before and beforeEach, check the values added to this and store the state. Anything set by a before is guaranteed to exist, as is, during the next test. This is a horrible idea, don't do it unless you don't want to use Proxies (please don't do this).
• Replace this with a Proxy. If a before block is executing and a value is added to this, store the value in one object (let's call it staticThis). If a beforeEach is executing and a value is added to this, store the value on another object (let's call it localThis). When a test (or another hook) tries to access a value on this, the Proxy will intercept the reference and search both staticThis and localThis, returning the appropriate value. Before the next test begins, localThis is reset, but staticThis maintains state 😨. This might end up looking something like this:

const staticThis = {};
let localThis = {};

this = new Proxy({}, {
  set:            (target, property, value) => {
    if(magic.isNonEachHook() || magic.isTestCase()) {
      localThis[property] = value;

      return;
    }

    staticThis[property] = value;
  },
  get:            (target, property) => {
    if(localThis[property]) {
      return localThis[property];
    }

    return staticThis[property];
  },
  has:            (target, property) => {
    return !!(localThis[property] || staticThis[property]);
  },
  deleteProperty: (target, property) => {
    delete localThis[property];
    delete staticThis[property];
  },
  defineProperty: (target, property, descriptor) => {
    if(magic.isNonEachHook() || magic.isTestCase()) {
      return Object.defineProperty(localThis, property, descriptor);
    }

    return Object.defineProperty(staticThis, property, descriptor);
  },
  ownKeys:        (target, property) => {
    return Object.getOwnPropertyNames(staticThis).concat(Object.getOwnPropertyNames(localThis));
  }
});

If backwards compatibility is a must, I like the second idea better. Of course, it does leave some room for discussion. For example, how would you handle a beforeEach hook resetting a value set by a before hook? The method that seems the most intuitive and expected is that staticThis's value would be reset. However, backwards compatibility would likely dictate that the value is simply added to localThis and the Proxy simply prefer values on localThis over those defined on staticThis.

Of course, changing something like this will be backwards breaking for some people no matter what (unless we don't actually fix the problem). Those that expect the prototypical this and have taken advantage of it (poor, poor prototypical this), will have side effects. I think the goal should be to minimize growing pains, not eliminate them (if we must do that much).

I hope all that makes sense. I'll be happy to clarify if necessary.

P.S. Please do not take my jokes (imagery) as anything more than that. If I have to write comments, I want to make it fun for the both of us.

[ScottFreeCode]

Ok, I looked it up -- what I had seen before (no pun intended) was use of let in RSpec to shadow variables that are used in an outer suite's before(:each) instead of putting an identical-except-for-those-variables before(:each) inside the nested suite (and I guess in Mocha's case, um, doing something to disable the outer one when in the inner suite?). I don't really know how RSpec implements that under the hood and makes it look so much like a local variable to the hooks and tests, and frankly in JS I'm not sure there would be a way at all except for this, so my first thought was that you could simply use this variables set in before (non-Each) -- only, the prototype system foils that (the outer Each doesn't see the inner suite's this variables). (Tangentially, it was investigating whether this could be put to that use case that I figured out what Mocha's actual behavior is and how useless it is over var in a suite; the investigation took enough time and attention that I didn't remember to figure out whether RSpec's let differs from non-Each before hooks aside from some fancy optimizations. That investigation and my correcting my own understanding also derailed the thread enough that I'm not eager to link to it for sake of staying on topic, hence the attempt to summarize here.) So, as I understand it the options to achieve that use case are A) go to great lengths to recreate RSpec's let in JS or B) since this needs fixing (or at any rate being enhanced to not be useless) for other reasons, fix it in a way that the this it already kinda shares between all hooks is usable in a manner similar to nested let from RSpec. I dunno if that justifies it or not, but that was the context (no pun intended there either) that I was working off of. (And, relatedly, I dunno that we can stop people from mutating variables in a nested suite if we want to -- it's JavaScript, the language that's Functional in the sense of having first-class closures and such but not Functional in the sense of enforcing purity.)

code code Proxy code code

To clarify, in my mind, the question "can we even fix this without breaking other Mocha suite-nesting behavior to which it may be coupled" is more important than... uh, anything I can imagine a Proxy being useful for in this case; I don't even know whether Proxy is available in all the environments we support at this point.

something about backwards compatibility and implied medieval cruelty

I'm afraid there's been a misunderstanding -- obviously to "fix" the behavior is going to change the behavior and that will break anything that was relying on the aspects that were changed. But if the fix ever happens, we're going to get multiple/duplicate GitHub issues saying "You broke my suite context in Mocha ! Please fix this to the way it was right before!" (Believe me, I wish I were being cynical and exaggerating, but I'm actually speaking from experience.) Saying "Well we only broke half these things instead of all of them" isn't going to matter. What matters is that we need a more helpful response than "go learn the One True Way to write tests! issue closed!" We need to be able to give them a clear but detailed upgrade path and a succinct but not arbitrary understanding of why it's worthwhile.

I hope all that makes sense. I'll be happy to clarify if necessary.

Use all the flamboyant imagery you want, if there's a reason to do things one way and not another I would want to hear it; but I have no interest in picking sides in some war over which exact usage of testing tools is the One True Way Just Because the Alternative Happens to Involve State That Isn't Reset Between Tests (hey, if you want to go that far, why not just ban non-Each hooks -- just because it isn't in variables doesn't mean what they set up isn't state), or whatever other subtle difference I'm missing between #1195 and #3109. I'd rather just keep telling people that Mocha's this wasn't meant to be used that way and no it wasn't well designed than get involved in "Oh hey you were obviously trying to enhance this but you done screwed up 'cause I says so!" My interest in championing a change is prompted by the fact that the current behavior could be ditched without actually losing anything (provided we have a clear way to walk people through it), but it's also directly proportionate to whether it's reasonably agreed to be the right change -- hence my holding off on getting too detailed till I know what expectations other similar JS test runners are setting.

[c1moore]

No need to get defensive, just trying to add my 2-cents and a few suggestions in a funny manner. I will attempt to tone down the humor.

uh, anything I can imagine a Proxy being useful for in this case; I don't even know whether Proxy is available in all the environments we support at this point.

That's a fair point. Proxies weren't supported until Node v6. However, you could always use a polyfill.

The point of proxies was to allow support for what (I thought) you were asking: some sort of backward compatibility. By using a Proxy and splitting this into two objects, you can maintain the state set in before blocks, but completely reset data set in a beforeEach block. No more need for prototypical this and the problems associated with it. I even provided suggestions to mimic the current behavior of this even closer handling how overwriting a property in beforeEach could be handled. My first suggestion is more or less a similar approach; however, it avoids proxies. I mention I believe it should be avoided because its inefficient and potentially error-prone (i.e. it requires more work and more conditional logic), but the affect is the same.

However, I may have misunderstood what was being asked. In which case, that suggestion can be safely ignored.

something about backwards compatibility and implied medieval cruelty

That's not what I said ¯\_(ツ)_/¯

But if the fix ever happens, we're going to get multiple/duplicate GitHub issues saying "You broke my suite context in Mocha ! Please fix this to the way it was right before!" (Believe me, I wish I were being cynical and exaggerating, but I'm actually speaking from experience.)

Again, a valid point. But, that's what major version bumps are for: breaking changes. If this is too big of a change for anybody, they can simply lock their version into the current major version until they can update their code. It's not ideal, but it happens all the time: Angular, WebPack, Mongoose, etc.

In the meantime, educational material should be provided for those that are willing to update.

"go learn the One True Way to write tests! issue closed!"

I would hope the comment would be more helpful. Again, my comment was meant to be playful and exaggerate things. However, it is widely accepted that tests should not 1) be order dependent and 2) maintain state. The former is a requirement for the latter. (I was going to add resources, but I'm much to busy to find them right now for a feature I have a feeling won't be pursued.)

(hey, if you want to go that far, why not just ban non-Each hooks -- just because it isn't in variables doesn't mean what they set up isn't state)

State itself within a test isn't bad. Maintaining state across tests can cause dependencies between tests. Back in my younger days when testing was new to me, my tests were not the best. Many had to be executed in a specific order to set up state for the next test. Now I know why this is bad. One very simple and obvious reason this is bad is that if one of those tests become irrelevant and is removed or simply has to be updated, all tests that come after it now fail. Why do they fail? Not because what they are testing failed, but because state was not maintained. In other words, state across tests couples tests and coupling = bad.

The danger of Mocha's this isn't necessarily that it inherently maintains state, it's that how it maintains state is misunderstood/unexpected and causes unexpected bugs and dependencies. If these dependencies aren't detected, tests could pass/fail for reasons not even being tested.

I'd rather just keep telling people that Mocha's this wasn't meant to be used that way and no it wasn't well designed

That's a potential solution. It's the "stick your head in the sand" solution, but it is completely valid for Mocha to choose that route. Many frameworks have made decisions that I (and others) disagree with, but guess what, the world keeps turning. Frameworks are meant to be opinionated. If Mocha decides this is one place it takes a stand, more power to you. However, if you decide to go this route, please document the behavior of this explicitly and very clearly. Provide examples of places where this may not behave the way people expect it. You seem to get as many support requests for this (pun intended) as you would for a breaking change. This seems to suggest that Mocha should 1) fix the problem or 2) provide better documentation.

That being said nobody is saying Mocha is a bad testing framework (I'm certainly not). There's no need to get salty or take what I said personally. What we're saying is we believe we found a bug that may, at one time, have been a feature. We feel the environment has changed and Mocha should change, too. It's perfectly fine for Mocha to decide not to pursue our suggestions; however, if that is the case you need a "clear way to walk people through" this unexpected behavior.

Again, Mocha is awesome. It's a very intuitive testing framework. The number of people that use it is a testament to that.

[moll]

As the official poster I must admit I don't understand what you all are on about. :P Mountains of text and solutions that span from overengineered Proxies to cloning the world.

@c1moore:

No more need for prototypical this and the problems associated with it.

What problems? No-one has since talked about mutating the objects in this. We're talking about mutating this itself (shallow) to add or replace properties from outer scopes. That's it. Any other problem is not possible to solve because you end up having to do a deep clone and some of those nested objects are guaranteed to be unclonable (from db connections to identity objects).

@c1moore:

@moll:

Just to remind people, I think I covered the proper implementation of this 3 years ago in #1195. :)

I disagree. Please see #3109 I opened a few weeks ago. Prototypical this really doesn't solve the problem of context remaining between tests.

It solves your example (and mine at the top of this issue) exactly. What part of it doesn't it solve?

[c1moore]

It solves your example (and mine at the top of this issue) exactly. What part of it doesn't it solve?

I was told this was a duplicate of the issue I opened (#3109), which describes the problem my developers are facing with the current solution. If this is not the case, these comments should be placed on that issue. If the solution fixes the problems, why is the issue still open?

Regardless, I'm going to just have my developers use a var instead of this. Instead of going back and forth over this issue in a non-constructive manner, I am also unsubscribing from these issues. Thank you for your time and consideration.

[moll]

@c1moore: Just to be clear, no proposed solution is in Mocha's code base today AFAICT, so that explains both you and me experience these problems.

[boneskull]

The use of this for context is problematic in and of itself, regardless of any unexpected weirdness with the capital-C Context.

Given the keystrokes spent on this topic in this and various other issues and PRs, this needs to be distilled to its essence. From what I can gather, there are two main problems with Context:

• "Each" hook Context is coupled to its suite; all hooks should execute in series using the same Context
• Tests share context inappropriately

Does that sound right to everyone?

---

We can use a cigtm-like tool to see just how bad this would break things. If it looks like the impact will be minimal, we could do a major bump for it.

But if it's obviously going to break a lot of tests, then no, there is not enough justification nor urgency.

"Better docs" is more urgent in either case.

I don't really have the time to do the legwork, but if we can assert this won't break the tests of major open-source consumers of Mocha, that'd be a good start. This would require an implementation first, of course!

[moll]

@boneskull: That seems to match my understanding.

When these issues get addresssed, I'd also ensure the context handover from before to beforeEaches be aligned with them. The befores should run once and recursively with child groups (describes) inheriting from their parents. At the end, that object should in turn be inherited freshly for each test case, run through beforeEaches and finally passed to the actual test. If that's unclear, I could write an automated test like I did for this issue at the top there.

[boneskull]

@moll can you please write that example test, just to be sure?

[moll]

Will do.

[eexit]

Hello,

I haven't read the whole thing but I found a case when using https://github.com/dareid/chakram, test isolation fails. If, within a describe, I have a test failure, all following tests are failing as well.

I use a beforeEach to set up some chakram configuration like HTTP headers or so but each test is doing its own HTTP request and I don't understand how one failure could cause following ones to fail as well although they don't rely on it.

Could you advise please? For now, the only solution I found to efficiently help debug is to use the the --bail option so when one is failing, we focus on the first failure.

Thanks!

[boneskull]

@eexit without code, it's tough to help.

[eexit]

Hey @boneskull,

Sorry but I didn't find the time to try to reproduce out of my project. Since it's a proprietary code, I cannot share nothing.
If I find some time to reproduce in a simpler scope, I'd gladly share my findings.