Fixture returns outdated/false data

[andreasremdt]

Current behavior:

Reading and writing fixtures seems to not work as expected (please let me know if this is my error). Across two different tests (within the same spec) the returned value from cy.fixture is outdated and should have been updated by a previous call to cy.writeFile. This looks to me like a caching issue?

Desired behavior:

It should always return the latest data from the fixture and not something outdated.

Steps to reproduce: (app code and test code)

1. Create a empty spec and paste the below code.
2. Run the spec and see the output in the dashboard.

describe('fixture', () => {
  it('step 1', () => {
    // Create the fixture first
    cy.writeFile("cypress/fixtures/test-temp.json", {
      id: 1,
      name: 'Step 1'
    });

    // Let's see the data, it should be fine
    cy.fixture("test-temp").then(data => cy.log(data));

    // Update the fixture again
    cy.writeFile("cypress/fixtures/test-temp.json", {
      id: 1,
      name: 'Step 2'
    });
  });

  it('step 2', () => {
    // Let's wait 5 seconds just to be sure
    cy.wait(5000);

    // The returned data is { id: 1, name: 'Step 1' }
    // Should be { id: 1, name: 'Step 2' }
    cy.fixture("test-temp").then(data => cy.log(data));
  });
});

Versions

Cypress 3.4.0 & 3.3.2
MacOS Mojave
Chrome 75

UPDATE: No need to create 2 different tests, it also happens inside the very same test.

[jennifer-shehane]

Yeah, this looks like a bug to me - if it's not, then it's really unexpected behavior.

I tried using an alias to reference the fixture and also tried using ugly thenables through the Cypress chains to ensure they were being run after another - it always references the original content of the fixture, even though I can see that the content of the fixture has changed.

Reproducible failing test

describe('fixture', () => {
  it('step 1', () => {
    // Create the fixture first
    cy.writeFile('cypress/fixtures/test-temp.json', {
      id: 1,
      name: 'Step 1',
    })

    // Let's see the data, it should be fine
    cy.fixture('test-temp').then((data) => {
      cy.log(data)
      expect(data.name).to.eq('Step 1')
    })

    // Update the fixture again
    cy.writeFile('cypress/fixtures/test-temp.json', {
      id: 2,
      name: 'Step 2',
    })

    cy.fixture('test-temp').then((data) => {
      cy.log(data)
      expect(data.name).to.eq('Step 2')
    })
  })
})

[WolfeLogic-LHR]

Maybe I'm chaining functions with the expectation of a void promise improperly, but I'm running into the same issue. I'm expecting a fixture to read from a file after the file's been updated.

I produce a "random" test ID with a before() hook

Cypress.Commands.add('randID', () => {
	var text = ""
	var possible = "abcdefghijklmnopqrstuvwxyz0123456789"

	for (var i = 0; i < 10; i++) {
		text += possible.charAt(Math.floor(Math.random() * possible.length))
	}
	cy.writeFile('cypress/fixtures/testID.json', { testID: text }).then((data) => {
		expect(data.testID).to.equal(text)
		cy.fixture('testID.json').then(data => {
			cy.log('TEST ID AFTER WRITE: ', data.testID)
		})
	})

})

Where testID.json ends up with something like:

{
  "testID": "okgt1hp768"
}

I run the actual test and end it with an after() hook:

Cypress.Commands.add('clearTestID', () => {
	let localTestID = ''
	cy.writeFile('cypress/fixtures/testID.json', { testID: " " }).then((data) => {
		expect(data.testID).to.equal(" ")

	})
	cy.wait(3000).then(() => {
		cy.fixture('testID.json').then(data => {
			localTestID = data.testID
			if (localTestID !== " ") {
				cy.wait(4000).then(() => {
					cy.fixture('testID.json').then(data => {
						cy.log('TEST ID AFTER WIPE: ', data.testID)
					})
				})
			}
			else {
				cy.log('TEST ID AFTER WIPE: ', data.testID)
			}
		})
	})

The assertion expect(data.testID).to.equal(" ") passes, and the JSON file reads:

{
  "testID": " "
}

but my log from cy.clearTestID() reads:
TEST ID AFTER WIPE: , okgt1hp768

These

Cypress.Commands.add('clearTestID', () => {
	cy.writeFile('cypress/fixtures/testID.json', { testID: " " }).then((data) => {
		expect(data.testID).to.equal(" ")
	})
})
Cypress.Commands.add('checkForClearID', () => {
	cy.fixture('testID.json').then(data => {
		cy.log('TEST ID AFTER WIPE: ', data.testID)
	})
})

Called as:

after(() => {
		cy.clearTestID().then(() => {
			cy.checkForClearID()
		})
	})

Also do not work.

And again... There's a more-than-reasonable chance I don't understand Cypress' asynchronicity as it relates to promise chaining (or at all?), but I thought I did? Happy to be schooled. But otherwise this is a pretty annoying bug... Or pattern.

[jennifer-shehane]

So I asked the team about this. We are indeed caching the reference to the fixture to save memory. This was intended at the time, but is likely not a great solution now.

We likely need to rework this logic, to not use the cache when the file has been changed. We can detect that the file has changed by initially getting the SHA, compare if SHA has changed, otherwise use cache. Alternatively look at modifiedAt time of the file (if works on all platforms)

Workaround:

You can use cy.readFile() instead of fixture as this will not use the cacheing.

[WolfeLogic-LHR]

Thanks Jennifer. Knowing that cy.fixture() caches the reference and that cy.readFile() reads the current state of the file is a solid workaround.

[x-yuri]

Not only doesn't it detect fixture changes, it also ignores encoding when caching. Having said that, I need it (a fixture in different encodings) for a test repo, where I'm going to show several solutions to the same problem. Not for testing a site.

As for changing fixtures, I'm not sure it's often needed. @andreasremdt Can you elaborate your case? But adding ignoreCache and/or dontCache options would probably resolve the issue. That might be easier to implement than detecting file changes.

Another workaround (that shouldn't go without an explaining comment) is adding slashes:

cy.fixture('upload.png');
cy.fixture('/upload.png');
cy.fixture('//upload.png');

[mwren-mshanken]

@x-yuri

My use case is:

I am stubbing getting an array of data that pre-populates fields on a page, altering the data, then I am clicking a "save button", stubbing the new data into a new fixture, then reloading the page and stubbing in my new data. I do this in multiple tests to test things like clearing all data, populating fields, making sure header elements update based on new data values. This allows me to mock tests to both our get API and the API that updates this data.

An example of the function I call to do all this:

export function saveApplicationAndReload() {
  clickSaveButton();
      cy.wait('@updatedApplication').then((xhr) => {
        cy.writeFile('cypress/fixtures/updatedApplication.json', xhr.request.body.application).then(() => {
          cy.readFile('cypress/fixtures/updatedApplication.json').then(() => {
            cy.fixture('updatedApplication.json').as('updatedApplication')
            cy.route('GET', 'applications/*', '@updatedApplication').as('getUpdatedApplication')
          })
        })
      })
      cy.reload()
      cy.wait('@getUpdatedApplication')
}

Since the fixture file always returns the same, original value, this function only works properly the first time it's called.

Replacing this with a cy.readFile() call works for now, but it took me a lot of time to track down this issue, and find the workaround due to the current implementation.

[nephix]

If you don't want to deal with workarounds and your fixtures rarely change then you can also remove the following directories:

• MacOS: ~/Library/Caches/Cypress
• Linux: ~/.cache/Cypress
• Windows: /AppData/Local/Cypress/Cache

And run in your project folder rm -rf node_modules && npm i
This will cause Cypress to reinstall, but much better than introducing workarounds in many of your tests IMO

[x-yuri]

@mwren-mshanken So you use one fixture for many tests and many purposes? That's not how fixtures are to be used, don't you agree? The solution is probably to use a different fixture name in different tests.

Speaking of the bigger picture... you do an xhr request, save the body in a fixture, and then stub subsequent requests with the fixture? Is it for performance reasons? Does using fixtures in such a way improve performance a lot?

Since the fixture file never gets updated, this function only works properly the first time it's called.

By the way, the fixture file is always updated, it's just that the cy.fixture() always return the first version of a file.

@nephix cy.fixture returns stale data if a fixture changes, what deleting Cypress has to do with it?

[mwren-mshanken]

I have multiple tests that interact with the same data object. In our application loading a page requires the same data to pre-populate fields, then we test various front-end functionality. Part of these tests includes what happens when we alter data, save the changes, then reload the page. This is why we need to:

• Know what the new data is when we save the page
• Store this data
• Reload the page and use this new data

Writing this data when it changes simulates our network layer, which hits the update API, then write the new data to a database. Stubbing this improves performance and keeps with the recommended pattern of having one golden-path test that does full end-to-end testing on all network paths, and stubs that simulate our e2e patterns.

For more information:

https://docs.cypress.io/guides/guides/network-requests.html#Stub-Responses

Suggested Use
Use for the vast majority of tests
Mix and match, typically have one true end-to-end test, and then stub the rest
Perfect for JSON APIs

[nephix]

@nephix cy.fixture returns stale data if a fixture changes, what deleting Cypress has to do with it?

Looks like cypress somehow caches them in the mentioned directories and deleting them therefore "cleares the cache"

[x-yuri]

No, the cache is in memory, as said above.

[nephix]

Definitely false, as the outdated fixture persists across sessions. It's probably serialized and stored somewhere, which is then re-initialized on a new session.

[x-yuri]

@nephix You seem so confident. Can you possibly back up your words? I can't reproduce what you're saying:

The test:

    it('...', () => {
        cy.fixture('1.txt').then(content => {
            cy.log(content);
            cy.writeFile('cypress/fixtures/1.txt', String(Number(content) + 1));
        })
        cy.fixture('1.txt').then(content => {
            cy.log(content);
        });
    });

$ git clone -b cypress-change-fixture https://github.com/x-yuri/issues cypress-change-fixture
$ cd cypress-change-fixture
$ npm i
$ npx cypress open

Then choose 1.spec.js, and you'll see:

Then press "r" (Run All Tests), and you'll see:

Which means fixtures don't persist across sessions. In case they persist for you under some circumstances, please provide exact and easy to follow instructions like I did.

[nephix]

@nephix You seem so confident. Can you possibly back up your words?

Yes have done it several times already since I've posted and wouldn't have posted it if it didn't work for me.

Which means fixtures don't persist across sessions.

Definitely not my observation.

In case they persist for you under some circumstances, please provide exact and easy to follow instructions like I did.

Can't really provide you with a minimal example from my phone, but in my test cases I've initialized them via:

cy.fixture('foo.json').as('foo')

and this.foo always had the same values from the first test run, even though I have changed content in foo.json. Could mean that .as() does some additional magic. Removing cypress and reinstalling it through npm i solved it for me though

provide exact and easy to follow instructions

I already provided instructions above that solve the problem of loaded fixtures being outdated and just wanted to provide another option besides the workaround mentioned here

[x-yuri]

Can't really provide you with a minimal example from my phone

I'm not asking to answer right away, please take your time to make Cypress better.

in my test cases I've...

I've once again followed your instructions, and still can't reproduce. Probably because your instructions were not exact. Can you please spend some time and give us a sure way to reproduce your issue? Ideally, create a repository one can clone, npm i, cypress open and see the result.

I already provided instructions above that solve the problem of loaded fixtures being outdated

Let's make it clear. The issue you're experiencing is not the issue described in the original post. Because your solution doesn't solve the original issue. Although both fall under the "outdated fixture data" category. That's why I'm trying to make you share the exact steps needed to reproduce your issue.

[nayakavik]

Thanks for this thread. Wasted a lot of time on this due to the fact that I am not a JS expert :) I could clearly see in the console in runner that the contents of the file under fixtures folder is getting updated but my assertion was failing due to cached data of the original content of the fixture.
cy.readFile() is a real savior.

I am not sure whether dynamically changing the data of fixture files is a best practice or not - still I hope the bug is fixed soon.

[x-yuri]

@nayakavik

I am not sure whether dynamically changing the data of fixture files is a best practice or not

L'et's try to find out, why do you need it? Describe your case?

[ngokevin]

I had a use case where I needed to change my fixtures to match changes in responses to my API routes, or I am just actively building test data (versus a DB dump).

workaround:

Cypress.Commands.add('Intercept', (method, route, fixture) => {
  return cy.readFile(`cypress/fixtures/${fixture}`).then(contents => {
    return cy.intercept(method, route, {body: contents}); 
  });
});

[x-yuri]

I don't really see a case for changing fixtures. Fixtures is something prepared in advance and shouldn't change. If you want to randomize test data, you need faker or something.

Change fixtures to match changes made by API requests? Do you even need fixtures in this case? To answer that question I need more details. What data you have in fixtures? What requests you make?

The bottom line is if you need to change fixtures, that's most likely a sign you're doing something wrong. Or so I think for now.

[ngokevin]

Your database or API responses never change, at all? The maintainers have flagged it as a bug, I'm not sure why you are filibustering here. Anyways, I've added my workaround to try to be actually helpful.

[x-yuri]

@ngokevin

Your database or API responses never change, at all?

So instead of changing the fixtures before running the tests you're changing them while running the tests?.. Doesn't that ring a bell? Doesn't the word "fixutre" mean something fixed, not changing?

The maintainers have flagged it as a bug, I'm not sure why you are filibustering here.

Did you consider that developers look at the issue and think, "Yeah, kind of unexpected. But well, why would they do that? There are more important issues to solve. We'll get back to it later." So instead of filibustering, why don't you put some effort, provide more details, explain why you can't do it without changing the fixtures, so that everybody (not just you) could understand that yes, indeed, a legitimate case. That would be definitely helpful for people solving the Y problem.

And I suggest you read this: https://xyproblem.info/

P.S. And then they complain that issues stay open for years...

[jennifer-shehane]

Please see our team's opinion on this issue here: #4716 (comment)

We do think the original issue would be valuable. But also, as mentioned above, there is a workaround and this falls below some other priorities we have at the moment.

We would be open to a PR to change this behavior so that fixture references are not cached.

[Xvier]

@x-yuri, a fixture does not mean or state that it should be only made before testing.
If you even read the cypress docs of cy.writefile() you will see an example of calling an API, saving that data and use it somewhere in your test.

It's a really valid use-case if you need to use data from a database to store it somewhere.

https://docs.cypress.io/api/commands/writefile.html#Write-response-data-to-a-fixture-file

[Shravandoe]

I tried using the read file function to read the JSON data. Unlike cy.fixture, this object returned by read file isn't available across multiple steps. Is there a way to handle it ?

Below is the code that worked

beforeEach(function()
      {
        cy.fixture('Staffnews_Corporate.json').then((testData)=>{
         this.testData = testData
        });
      })

However the data returned wasn't the latest data. So I changed it to below

    beforeEach(function()
      {
        cy.readFile('cypress/fixtures/eoicorporate.json').then((testDataObject)=>{
       const  testData = testDataObject;
         });

This does save the data in testData object , I can see that on console using debuger. However it's not available to the steps in test ie it block.

  it('Submit Corporare EOI', () => {
      
      cy.visit(testData.url);
  }

This is the error i'm getting

[sagarqapomelo]

I am still facing this issue . i am writing my response to fixture and when trying to invoke it returns OLD Values

[alundiak]

OMG. I also spent lot of time with Cypress, and I really can name it as a "waste of time" :) because I didn't look to my issue in regards to cy.fixtures() - because I believed code is correct. Only extensive, deep debugging helped me to understand/see what is exactly issue under the hood of the Cypress weird behavior.

Despite the fact @andreasremdt Cypress version is 3.4.0 & 3.3.2 I use version 5.6.0 on Windows and on Ubuntu (CI) together with NodeJS v14.x and npm v6.x. And this cy.fixture() really CACHES content by default. Now, reading all comment above, and realizing that purpose of fixtures is to give a static, assumably not changed data for our tests, it's expected that huge JSON files (in my example) will be and must be cached.

So yes, as @jennifer-shehane suggested, and many other already have used/proved, cy.readFile() really takes the latest content of file.

PS Actually problem here is much wider I assume. I got project code where for some reason cy.fixture() used as approach to read content from dynamically/frequently changed JSON file. In theory it's assumed OK to have login credentials as static data, but in my case, it's cookies as content were changed all the time, and new values written to file on runtime for every describe() or it() by dedicated script relying on NodeJS fs.writeFileSync(). I didn't see the problem at first, but later on, weeks of wasted time :) simply to realize, that approach to read file was selected wrong at the beginning. Now I iterate over result from cy.readFile(), and actually I had similar idea to try - I intended to use NodeJS fs.readFileSync() but wasn't able to run it in non-NodeJS env easy (yes, with no browserified nodejs).

[martinMMO]

Can someone please explain how to use cy.readFile() workaround mentioned by @jennifer-shehane ? I can verify values with this command, but how can I use it as variable in next step/scenario ?

[ukbw]

@martinMMO I was facing the issue where I was reading cy.fixture(fixtureFileName) and it was reading the cached value so I replaced cy.fixture to cy.readFile(./cypress/fixtures/${fixtureFileName}) and it worked.