Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementing possibility to attach files to the json reports inside the afterEach hook #1089

Closed
3 tasks done
artem-sokorskyi opened this issue Sep 2, 2023 · 17 comments
Closed
3 tasks done

Comments

@artem-sokorskyi
Copy link

Current behavior

attach fn will fail with error message if invoked inside afterEach.
> Unexpected state in createStringAttachmentHandler: test-finished (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)

Desired behavior

Attaching successfully the file to the json reports inside the afterEach hook. I suppose the attached file could go under the last executed step of the scenario in the json report

Test code to reproduce

dummy-hook.js

import { attach } from '@badeball/cypress-cucumber-preprocessor';

const catchFailedTestcase = (done) => {
  try {
    done();
  } catch (err) {}
};

afterEach((done) => {
  const test = Cypress.mocha.getRunner().test.ctx.currentTest;
  // const caseStartTimeUnixMs = test.wallClockStartedAt.getTime();
  // hardcoded the unix timestamp to be earlier added to the "cypress/downloads" directory than your scenario will be executed
  const caseStartTimeUnixMs = 1693648794962;
  const { platform } = window.Cypress;
  const downloadFolderGlobPath = `${Cypress.config('downloadsFolder')}/**/*.*`;

  cy.task('getFileSinceTime', {
    globFolderPath: downloadFolderGlobPath,
    platform,
    startingFromTimeUnixMs: caseStartTimeUnixMs,
  }).then((downloadPathes) => {
    const availableMimeTypes = {
      csv: {
        mimeType: 'text/csv',
        encoding: 'utf-8',
      },
      jpeg: {
        mimeType: 'base64:image/jpeg',
        encoding: 'base64',
      },
      jpg: {
        mimeType: 'base64:image/jpeg',
        encoding: 'base64',
      },
      json: {
        mimeType: 'application/json',
        encoding: 'utf-8',
      },
      png: {
        mimeType: 'base64:image/png',
        encoding: 'base64',
      },
      pdf: {
        mimeType: 'base64:application/pdf',
        encoding: 'base64',
      },
      txt: {
        mimeType: 'text/plain',
        encoding: 'utf-8',
      },
      xls: {
        mimeType: 'application/vnd.ms-excel',
        encoding: 'utf-8',
      },
      xlsx: {
        mimeType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
        encoding: 'utf-8',
      },
      xml: {
        mimeType: 'application/xml',
        encoding: 'utf-8',
      },
      zip: {
        mimeType: 'application/zip',
        encoding: 'utf-8',
      },
    };
    downloadPathes.forEach((downloadedItem) => {
      const fileExtension = downloadedItem.match(/\.(.+)$/)[1];

      const { mimeType, encoding } = availableMimeTypes[fileExtension];
      cy.readFile(downloadedItem, encoding).then((content) => {
        attach(content, mimeType);
      });
    });
  });
  cy.then(() => catchFailedTestcase(done));
});

cypress.config.js (remember to disable the "trashAssetsBeforeRuns" option inside the config object)

import path from 'path';
import { globSync } from 'glob';


const getRelativePathFromAbsolute = (absolutePath) => path.relative(process.cwd(), absolutePath);
const getUnixPathFromWin = (winPath) => winPath.split(path.sep).join(path.posix.sep);
  on('task', {
    getFileSinceTime({ globFolderPath, platform, startingFromTimeUnixMs }) {
      const relativeGlobPath = getRelativePathFromAbsolute(globFolderPath);
      const parsedGlobFolderPath = platform.includes('win') ? getUnixPathFromWin(relativeGlobPath) : globFolderPath;
      return globSync(parsedGlobFolderPath).filter(
        (name) => Date.parse(fs.statSync(name).ctime) - startingFromTimeUnixMs >= 0,
      );
    },
  });

Next steps:

  • Enable json reports in your preprocessor config
  • Put dummy pdf file into the "cypress/downloads" directory
  • Create a scenario with one simple step (no matter what logic, can be cy.wrap(1).should('eq', 1)
  • Execute the scenario in run mode
  • Pay attention to the error message described in the actual behaviour above

Logs

DevTools listening on ws://127.0.0.1:63318/devtools/browser/53533b1d-ab11-4706-8209-6ae1725ba69b
  cypress-cucumber-preprocessor resolved environment overrides {} +0ms
  cypress-cucumber-preprocessor resolved explicit user configuration {
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: undefined },
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false
  cypress-cucumber-preprocessor } +1ms
  cypress-cucumber-preprocessor resolved configuration {
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: 'cucumber-report.html' },
  cypress-cucumber-preprocessor   pretty: { enabled: false },
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false,
  cypress-cucumber-preprocessor   implicitIntegrationFolder: '/'
  cypress-cucumber-preprocessor } +0ms

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        12.17.3                                                                        │
  │ Browser:        Chrome 116 (headless)                                                          │
  │ Node Version:   v18.17.1 (/usr/local/bin/node)                                                 │
  │ Specs:          1 found (Login.feature)                                                        │
  │ Searched:       cypress/e2e/features/User Identification/Login.feature                         │
  │ Experiments:    experimentalInteractiveRunEvents=true,experimentalOriginDependencies=true      │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

  cypress-cucumber-preprocessor beforeRunHandler() +328ms
  cypress-cucumber-preprocessor resolved environment overrides {} +2ms
  cypress-cucumber-preprocessor resolved explicit user configuration {
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: undefined },
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false
  cypress-cucumber-preprocessor } +0ms
  cypress-cucumber-preprocessor resolved configuration {
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: 'cucumber-report.html' },
  cypress-cucumber-preprocessor   pretty: { enabled: false },
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false,
  cypress-cucumber-preprocessor   implicitIntegrationFolder: '/'
  cypress-cucumber-preprocessor } +0ms

────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  Login.feature                                                                   (1 of 1)
  cypress-cucumber-preprocessor beforeSpecHandler() +1s
  cypress-cucumber-preprocessor resolved environment overrides {} +48ms
  cypress-cucumber-preprocessor resolved explicit user configuration {
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: undefined },
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false
  cypress-cucumber-preprocessor } +0ms
  cypress-cucumber-preprocessor resolved configuration {
  cypress-cucumber-preprocessor   stepDefinitions: [ 'cypress/e2e/**/stepDefinitions/**/*.js' ],
  cypress-cucumber-preprocessor   messages: { enabled: true, output: 'jsonlogs/messages.ndjson' },
  cypress-cucumber-preprocessor   json: { enabled: true, output: 'jsonlogs/log.json' },
  cypress-cucumber-preprocessor   html: { enabled: false, output: 'cucumber-report.html' },
  cypress-cucumber-preprocessor   pretty: { enabled: false },
  cypress-cucumber-preprocessor   filterSpecs: true,
  cypress-cucumber-preprocessor   omitFiltered: false,
  cypress-cucumber-preprocessor   implicitIntegrationFolder: '/Users/artemSokorskyi/Desktop/Workspace/work projects/olympics-hrp-qa/cypress/e2e/features'
  cypress-cucumber-preprocessor } +1ms
  cypress-cucumber-preprocessor resolving step definitions using template(s) [ 'cypress/e2e/**/stepDefinitions/**/*.js' ] +0ms
  cypress-cucumber-preprocessor replacing [filepath] with 'User Identification/Login' +0ms
  cypress-cucumber-preprocessor replacing [filepart] with [ 'User Identification/Login', 'User Identification' ] +0ms
  cypress-cucumber-preprocessor for 'cypress/e2e/features/User Identification/Login.feature' yielded patterns [ 'cypress/e2e/**/stepDefinitions/**/*.js' ] +0ms
  cypress-cucumber-preprocessor found step definitions [
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/NavigateSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/GenericSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/DropdownSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/AssertFilteringSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/ApiSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/PageSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/MemorySteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/InputSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/ClickerSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/core/stepDefinitions/CheckSteps.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/Hooks/clearTestStateHook.js',
  cypress-cucumber-preprocessor   'cypress/e2e/project/stepDefinitions/Hooks/IntegrationHooks.js'
  cypress-cucumber-preprocessor ] +7ms


  Login
  cypress-cucumber-preprocessor specEnvelopesHandler() +2s
    - Successful login with valid uppercase/lowercase email (example #1)
    - Successful login with valid uppercase/lowercase email (example #2)
    - Login with invalid email format
    - Login with empty fields
    - Login with invalid credentials
  cypress-cucumber-preprocessor testCaseStartedHandler() +31ms
  cypress-cucumber-preprocessor testStepStartedHandler() +5ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +2s
  cypress-cucumber-preprocessor testStepStartedHandler() +4ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +3s
  cypress-cucumber-preprocessor testStepStartedHandler() +5ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +242ms
  cypress-cucumber-preprocessor testStepStartedHandler() +10ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +5s
  cypress-cucumber-preprocessor testStepStartedHandler() +31ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +112ms
  cypress-cucumber-preprocessor testStepStartedHandler() +5ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +575ms
  cypress-cucumber-preprocessor testStepStartedHandler() +25ms
  cypress-cucumber-preprocessor testStepFinishedHandler() +52ms
  cypress-cucumber-preprocessor testCaseFinishedHandler() +22ms
  cypress-cucumber-preprocessor createStringAttachmentHandler() +35ms
  cypress-cucumber-preprocessor afterScreenshotHandler() +120ms
    1) "after each" hook for "Dummy Scenario"


  0 passing (12s)
  5 pending
  1 failing

  1) Login
       "after each" hook for "Dummy Scenario":
     CypressError: `cy.task('cypress-cucumber-preprocessor:create-string-attachment')` failed with the following error:

> Unexpected state in createStringAttachmentHandler: test-finished (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)

https://on.cypress.io/api/task

Because this error occurred during a `after each` hook we are skipping all of the remaining tests.
      at <unknown> (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:151171:78)
      at tryCatcher (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:18744:23)
      at Promise._settlePromiseFromHandler (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16679:31)
      at Promise._settlePromise (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16736:18)
      at Promise._settlePromise0 (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16781:10)
      at Promise._settlePromises (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:16857:18)
      at _drainQueueStep (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13451:12)
      at _drainQueue (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13444:9)
      at ../../node_modules/bluebird/js/release/async.js.Async._drainQueues (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13460:5)
      at Async.drainQueues (https://dev-hrp.vpc.qcue.com/__cypress/runner/cypress_runner.js:13330:14)
  From Your Spec Code:
      at createStringAttachment (webpack:///./node_modules/@badeball/cypress-cucumber-preprocessor/dist/entrypoint-browser.js:94:0)
      at attach (webpack:///./node_modules/@badeball/cypress-cucumber-preprocessor/dist/entrypoint-browser.js:100:0)
      at Context.eval (webpack:///./cypress/e2e/project/stepDefinitions/Hooks/IntegrationHooks.js:280:14)
  
  From Node.js Internals:
    Error: Unexpected state in createStringAttachmentHandler: test-finished (this might be a bug, please report at https://github.com/badeball/cypress-cucumber-preprocessor)
        at createError (/Users/artemSokorskyi/Desktop/Workspace/work projects/olympics-hrp-qa/node_modules/@badeball/cypress-cucumber-preprocessor/dist/helpers/error.js:9:12)
        at createStringAttachmentHandler (/Users/artemSokorskyi/Desktop/Workspace/work projects/olympics-hrp-qa/node_modules/@badeball/cypress-cucumber-preprocessor/dist/plugin-event-handlers.js:434:43)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)



  cypress-cucumber-preprocessor afterSpecHandler() +1s
  Hook failures can't be represented in any reports (messages / json / html), thus none is created for cypress/e2e/features/User Identification/Login.feature.

Versions

  • Cypress version: 12.17.3
  • Preprocessor version: 18.0.4
  • Node version: 18.17.1

Checklist

  • I've read the FAQ.
  • I've read instructions for logging issues.
  • I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).
@hammzj
Copy link

hammzj commented Sep 6, 2023

See issue #1051

@badeball
Copy link
Owner

Sorry for taking so long to get back to you, @artem-sokorskyi and @hammzj. As mentioned, this has come up before. I was not entirely convinced of the use-cases, but adding downloads to the report seems akin to screenshots. Ideally I would have liked to have a download API in Cypress and have native support for adding downloads to reports, but based on previous experience I'd say there's little chance of ever seeing this added.

I've been trying to come up with the best way of supporting all of the use-cases so far, and I think having a after step handler in the node environment, providing an attach function, will suffice, Something along the lines of that shown below.

Adding logs from cypress-terminal-report (@hammzj :

await addCucumberPreprocessorPlugin(on, config, {
  onAfterStep({ wasLastStep, attach }) {
    if (wasLastStep) {
      const logs = ""; // Retrieve logs from  cypress-terminal-report.

      attach(logs);
    }
  }
});

Adding downloads (@artem-sokorskyi):

await addCucumberPreprocessorPlugin(on, config, {
  onAfterStep({ wasLastStep, attach }) {
    const downloads = []; // Retrieve recent downloads.
    
    for (const download of downloads) {
      attach(download);
    }
  }
});

Adding resized screenshots (@MDG-JHowley):

await addCucumberPreprocessorPlugin(on, config, {
  async onAfterStep({ wasLastStep, attach }) {
    const screenshots = []; // Retrieve recent screenshots.
    
    for (const screenshot of screenshots) {
      attach(resize(screenshot));
    }
  }
});

@artem-sokorskyi
Copy link
Author

cool stuff!!!

@MDG-JHowley
Copy link

@badeball yes this would be great - as you say, we really just need the attach method available in the node environment. onAfterStep seems like a good place to add that hook but what happens if the test fails? Would this hook still be run?

@badeball
Copy link
Owner

badeball commented Oct 23, 2023

but what happens if the test fails? Would this hook still be run?

This, contrary to others, can in fact be run despite test failures.

@hammzj
Copy link

hammzj commented Oct 24, 2023

So, onAfterStep is a node hook, but runs like this:

  • after:step event (Node env)
  • onAfterStep function (Cypress env)

If this is the case, this is really cool. I wonder, could this will also get around After hooks not running on test failures since it's able to call back into the browserified Cypress environment?

@badeball
Copy link
Owner

No, this would run purely in the node environment and it cannot call back into the browser environment.

badeball added a commit that referenced this issue Nov 2, 2023
This is necessary for EG. consistently adding downloads as attachments,
in the absence of a proper "after download api" [1]. I hope and believe
that this fixes #1089 [1].

[1] cypress-io/cypress#27779
[2] #1089
badeball added a commit that referenced this issue Nov 2, 2023
This is necessary for EG. consistently adding downloads as attachments,
in the absence of a proper "after download api" [1]. I hope and believe
that this fixes #1089 [1].

[1] cypress-io/cypress#27779
[2] #1089
@badeball
Copy link
Owner

badeball commented Nov 2, 2023

I've released the above-mentioned API with v19.0.0. You can read more about it here. I hope this gives you the ability to attach files "post test" (albeit as part of the last step, as there are no "test scoped" attachments in Cucumber).

@hammzj
Copy link

hammzj commented Nov 2, 2023

This is awesome! I'll need to test it out in the next week or so. Thanks for adding this

@MDG-JHowley
Copy link

@badeball is there anyway to pass the current test details into this hook?

@badeball
Copy link
Owner

What do you have in mind specifically?

@MDG-JHowley
Copy link

Well ideally the feature/spec and scenario/test names and whether the test has failed at that point.

I'm trying to attach logs (from cypress-terminal-report) along with screenshots/videos (I can't use the built in screenshot handler because I want to embed an html <img> tag referencing an s3 bucket where the images will ultimately reside) to the report.

For the logs and the videos I need to know the current Feature/spec. For the screenshots the local file path is based on feature/spec, scenario/test and test failure state.

My current hack is a custom cy.task in a before hook which writes a file I can later read from in the onAfterStep hook.

@badeball
Copy link
Owner

Would it help to have to same arguments as BeforeStep (as seen here), along with a result-parameter?

@MDG-JHowley
Copy link

Yes, that would be perfect

badeball added a commit that referenced this issue Dec 9, 2023
badeball added a commit that referenced this issue Dec 9, 2023
badeball added a commit that referenced this issue Dec 17, 2023
@badeball
Copy link
Owner

badeball commented Dec 19, 2023

I've changed the onAfterStep somewhat in v20.0.0. It is now invoked with the above-mentioned parameters. It is no longer invoked with wasLastStep. It is also no longer invoked after scenario hooks.

@MDG-JHowley
Copy link

MDG-JHowley commented Jan 2, 2024

I've changed the onAfterStep somewhat in v20.0.0. It is now invoked with the above-mentioned parameters. It is no longer invoked with wasLastStep. It is also no longer invoked after scenario hooks.

@badeball I've tested this and it is all working nicely, thank you. A minor typo in your docs though:

@badeball
Copy link
Owner

badeball commented Jan 2, 2024

A minor typo in your docs though:

Thanks for the heads up! I've fixed this with 29533f7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants