Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Percy snapshot command returns exit code 0 when build fails #1181

Open
IanVS opened this issue Feb 3, 2023 · 3 comments
Open

Percy snapshot command returns exit code 0 when build fails #1181

IanVS opened this issue Feb 3, 2023 · 3 comments
Labels
✨ enhancement New feature or request

Comments

@IanVS
Copy link

IanVS commented Feb 3, 2023

The problem

When my percy snapshots fail due to running out of credits or some other reason, my CI does not alert me, because the exit code returned is 0.

For example:

▶ npx percy snapshot dist/blog
[percy] Percy has started!
[percy] Snapshot taken: /internationalizing-docs-pages-with-astro/index.html
[percy] Failed to create build
[percy] Error: Missing Percy token
[percy] Build not created

▶ echo $?
0

You can see that the build failed (in this case because it couldn't find a token), but the process returned with an exit code of 0, which is supposed to mean everything worked correctly.

Environment

  • Node version: 16.16.0
  • @percy/cli version: 1.18.0
  • Version of Percy SDK you’re using:
  • If needed, a build or snapshot ID:
  • OS version:
  • Type of shell command-line [interface]: zsh

Details

A similar issue is discussed in #539, but the fix there only involved Cypress.

Debug logs

I tried not to trim this, but GitHub gave me an error (There was an error creating your Issue: body is too long (maximum is 65536 characters).), so I trimmed out some of the middle.

Debug logs

[percy:config] Found config file: .percy.yml (0ms)
[percy:config] Using config:
{
version: 2,
upload: {
files: '**/*.{png,jpg,jpeg,avif,webp}',
ignore: ''
},
snapshot: {
widths: [
375,
1280
],
minHeight: 1024,
percyCSS: ''
},
discovery: {
networkIdleTimeout: 750,
disableCache: true
}
} (19ms)
[percy:core:browser] Launching browser (25ms)
[percy:core:browser] Browser connected [23778]: HeadlessChrome/96.0.4664.0 (390ms)
[percy:core] Percy has started! (1ms)
[percy:core:snapshot] --------- (78ms)
[percy:core:snapshot] Received snapshot: /index.html (0ms)
[percy:core:snapshot] - url: http://localhost:62602/index.html (1ms)
[percy:core:snapshot] - widths: 375px, 1280px (0ms)
[percy:core:snapshot] - minHeight: 1024px (0ms)
[percy:core:snapshot] - discovery.allowedHostnames: localhost (0ms)
[percy:core:snapshot] - discovery.disableCache: true (0ms)
[percy:core:snapshot] --------- (1ms)
[percy:core:discovery] - Skipping remote resource (0ms)
[percy:core:discovery] Processing resource: https://stats.g.doubleclick.net/j/collect?t=dc&aip=1&_r=3&v=1&_v=j99&tid=UA-179012409-1&cid=89141066.1675434506&jid=1787100395&gjid=822920426&_gid=1002200174.1675434506&_u=YEBAAUAAAAAAACAAIC~&z=535542665 (57ms)
[percy:core:discovery] - Skipping remote resource (1ms)
[percy:core:discovery] Handling request: https://www.google.com/ads/ga-audiences?t=sr&aip=1&_r=4&slf_rd=1&v=1&_v=j99&tid=UA-179012409-1&cid=89141066.1675434506&jid=1787100395&_u=YEBAAUAAAAAAACAAIC~&z=1918529209 (0ms)
[percy:core:discovery] Processing resource: https://www.google.com/ads/ga-audiences?t=sr&aip=1&_r=4&slf_rd=1&v=1&_v=j99&tid=UA-179012409-1&cid=89141066.1675434506&jid=1787100395&_u=YEBAAUAAAAAAACAAIC~&z=1918529209 (112ms)
[percy:core:discovery] - Skipping remote resource (0ms)
[percy:core:page] Inject @percy/dom (504ms)
[percy:core:page] Serialize DOM (5ms)
[percy:core:page] Inject @percy/dom (4ms)
[percy:core:page] Serialize DOM (4ms)
[percy:core:discovery] Wait for 750ms idle (2ms)
[percy:core:page] Inject @percy/dom (1ms)
[percy:core:page] Inject @percy/dom (0ms)
[percy:core:discovery] Wait for 750ms idle (3ms)
[percy:core:page] Serialize DOM (0ms)
[percy:core:page] Serialize DOM (0ms)
[percy:core:page] Inject @percy/dom (1ms)
[percy:core:page] Serialize DOM (2ms)
[percy:core:discovery] Wait for 750ms idle (4ms)
[percy:core:discovery] Wait for 750ms idle (2ms)
[percy:core:discovery] Wait for 750ms idle (2ms)
[percy:core:page] Inject @percy/dom (48ms)
[percy:core:page] Inject @percy/dom (0ms)
[percy:core:page] Inject @percy/dom (0ms)
[percy:core:page] Inject @percy/dom (0ms)
[percy:core:page] Serialize DOM (3ms)
[percy:core:page] Serialize DOM (0ms)
[percy:core:page] Serialize DOM (0ms)
[percy:core:page] Serialize DOM (1ms)
[percy:core:discovery] Wait for 750ms idle (6ms)
[percy:core:discovery] Wait for 750ms idle (1ms)
[percy:core:discovery] Wait for 750ms idle (1ms)
[percy:core:discovery] Wait for 750ms idle (2ms)
[percy:core:page] Inject @percy/dom (156ms)
[percy:core:page] Serialize DOM (2ms)
[percy:core:discovery] Wait for 750ms idle (7ms)
[percy:core:page] Resize page to 1280x1024 @1x (508ms)
[percy:core:page] Resize page to 1280x1024 @1x (4ms)
[percy:core:page] Taking snapshot: /introducing-oidc-single-sign-on-for-admins/index.html @1280px (2ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/defined-og_kkrsT.avif (1ms)
[percy:core:page] Taking snapshot: /introduction-to-defined-networking/index.html @1280px (0ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:page] Resize page to 1280x1024 @1x (4ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/defined-og_kkrsT.avif (0ms)
[percy:core:discovery] - Skipping disallowed status [404] (1ms)
[percy:core:page] Taking snapshot: /blocklisting/index.html @1280px (1ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:page] Resize page to 1280x1024 @1x (1ms)
[percy:core:page] Resize page to 1280x1024 @1x (2ms)
[percy:core:page] Taking snapshot: /index.html @1280px (0ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/defined-og_kkrsT.avif (9ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/road-gate_ZsW3x0.avif (1ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/library_1Jzvth.avif (6ms)
[percy:core:page] Taking snapshot: /internationalizing-docs-pages-with-astro/index.html @1280px (0ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/defined-og_kkrsT.avif (2ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/road-gate_ZsW3x0.avif (0ms)
[percy:core:discovery] - Skipping disallowed status [404] (1ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/library_1Jzvth.avif (3ms)
[percy:core:discovery] - Skipping disallowed status [404] (1ms)
[percy:core:discovery] - Skipping disallowed status [404] (2ms)
[percy:core:page] Resize page to 1280x1024 @1x (32ms)
[percy:core:page] Resize page to 1280x1024 @1x (2ms)
[percy:core:page] Taking snapshot: /announcing-relay-support-in-nebula/index.html @1280px (0ms)
[percy:core:discovery] Wait for 750ms idle (1ms)
[percy:core:page] Resize page to 1280x1024 @1x (0ms)
[percy:core:page] Resize page to 1280x1024 @1x (1ms)
[percy:core:page] Taking snapshot: /newsletter-advanced-firewall-and-audit-logs/index.html @1280px (0ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:page] Taking snapshot: /newsletter-admin-api-cert-rotation-multiple-lighthouses/index.html @1280px (1ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:page] Taking snapshot: /relays-are-in-dn-today/index.html @1280px (1ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/radio-dishes_gicLf.avif (35ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/radio-dish-hero_Z1NeKLk.avif (2ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/auditlogs_29ls4A.avif (1ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/lighthouse_1JKmcn.avif (5ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/radio-dish-hero_Z1NeKLk.avif (6ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/radio-dishes_gicLf.avif (1ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/auditlogs_29ls4A.avif (1ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/lighthouse_1JKmcn.avif (1ms)
[percy:core:discovery] - Skipping disallowed status [404] (0ms)
[percy:core:discovery] - Skipping disallowed status [404] (1ms)
[percy:core:discovery] - Skipping disallowed status [404] (0ms)
[percy:core:discovery] - Skipping disallowed status [404] (0ms)
[percy:core:page] Resize page to 1280x1024 @1x (110ms)
[percy:core:page] Taking snapshot: /open-for-business/index.html @1280px (3ms)
[percy:core:discovery] Wait for 750ms idle (0ms)
[percy:core:discovery] Handling request: http://localhost:62602/assets/open_jieyQ.avif (27ms)
[percy:core:discovery] Processing resource: http://localhost:62602/assets/open_jieyQ.avif (17ms)
[percy:core:discovery] - Skipping disallowed status [404] (1ms)
[percy:core:page] Serialize DOM (467ms)
[percy:core:page] Page closed (9ms)
[percy:core:page] Serialize DOM (1ms)
[percy:core:page] Serialize DOM (0ms)
[percy:core:page] Serialize DOM (1ms)
[percy:core:page] Page closed (9ms)
[percy:core:page] Page closed (1ms)
[percy:core:page] Page closed (2ms)
[percy:core:page] Serialize DOM (14ms)
[percy:core:page] Page closed (12ms)
[percy:core:page] Serialize DOM (21ms)
[percy:core:page] Serialize DOM (1ms)
[percy:core:page] Serialize DOM (1ms)
[percy:core:page] Serialize DOM (0ms)
[percy:core:page] Page closed (5ms)
[percy:core:page] Page closed (4ms)
[percy:core:page] Page closed (2ms)
[percy:core:page] Page closed (4ms)
[percy:core:page] Serialize DOM (152ms)
[percy:core:page] Page closed (7ms)
[percy:core:browser] Closing browser (2ms)
[percy:core:browser] Browser closed (40ms)
[percy:core] Build not created (0ms)

Code to reproduce issue

N/A
@itsjwala
Copy link
Contributor

itsjwala commented Feb 3, 2023

Hi @IanVS 👋

  • I could see the build was not created, and thus build "fails" won't be correct.
    • If a build was created then you can use build:wait ref, it outputs useful information which you may process further.
  • Currently, it's by design that exit codes are managed this way, we'll evaluate the need for this in the future.
  • Though as a workaround, for this issue you may parse this log message and handle exits on your end.

@IanVS
Copy link
Author

IanVS commented Feb 3, 2023

Hm, but the point of the command is to perform a build and snapshot it, so how is failing to perform a build a successful outcome? :-/

The use case I really want to enable is to fail my CI when my snapshots run out or there's another problem. How would you suggest handling it? Does Build not created get logged out in all situations, and I should fail my CI based on that?

@itsjwala
Copy link
Contributor

itsjwala commented Feb 6, 2023

@IanVS

  • In current terminologies, the build is generally not created when PERCY_TOKEN is not passed, or the API has failed to acknowledge the request.
    • so Build not created log line can be used for this scenario.
  • for other scenarios where the build was created and for any reason didn't succeed can be evaluated usingbuild:wait ref.

@itsjwala itsjwala added the ✨ enhancement New feature or request label Feb 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
✨ enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@IanVS @itsjwala