Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sourcemaps upload with --url-prefix behavior is strange #1605

Closed
Memfisrain opened this issue May 9, 2023 · 7 comments
Closed

Sourcemaps upload with --url-prefix behavior is strange #1605

Memfisrain opened this issue May 9, 2023 · 7 comments

Comments

@Memfisrain
Copy link

Environment

We have sentry on-premise version 22.5.0
nodejs version 16
sentry-cli version 2.17.5

Steps to Reproduce

As part of releasing frontend applications, we create Sentry release, set commits, upload source maps and create deployment. The problem is that we have different environment and different CDN for stagings / preprod / prod.

When source maps uploaded first time to release, everything works good. Let's say, I deploy my code on stg25 , source maps are uploaded using the following command:

sentry-cli releases -o myOrg -p appjs-sdk files 1e2e77d8 upload-sourcemaps ./dist --url-prefix '~/stg25/appjs-sdk'

If I open appjs-sdk project in Sentry, then go to source maps configuration and select release 1e2e77d8 I will find source files together with source maps. Their names look like: ~/stg25/appjs-sdk/bootstrap-4b2cd0de.js, which is expected for me.

If after that, I decided to deploy this to preprod, source maps are uploaded using the following command:
sentry-cli releases -o myOrg -p appjs-sdk files 1e2e77d8 upload-sourcemaps ./dist --url-prefix '~/preprod/appjs-sdk'

If I open appjs-sdk project in Sentry, then go to source maps configuration and select release 1e2e77d8 I will not find sources and files with names ~/preprod/appjs-sdk/bootstrap-4b2cd0de.js which is unexpected for me. Even in Sentry tutorial video https://youtu.be/hEwpjRHRpC8?t=593 source maps are uploaded with different --url-prefix and they become visible on UI.

The problem is that we have a production issue in Sentry, and source maps are not applied to it.

Moreover, I see that some of our releases contain source maps and sources for many staging, preprod and prod, so it seems that the issue if flaky.

Expected Result

Uploading source maps with different --url-prefix results in showing them on UI under the release artifact and will apply to Sentry issues.

Actual Result

Uploading source maps with different --url-prefix seems has no effect after first source maps were uploaded with any --url-prefix

@kamilogorek
Copy link
Contributor

What's happening here I believe is that we are deduplicating files based on their content hash. If you want to have multiple files with exactly same content, use --no-dedupe flag for your upload comment, which should let it through.

If that still doesnt work, please provide debug logs with either --log-level=debug or SENTRY_LOG_LEVEL=debug.

@Memfisrain
Copy link
Author

@kamilogorek thanks for the response, however I still do not understand few things:

  1. How that source maps are stored on the server and in case I uploaded the same source maps but use different --url-prefix does it upload and store a duplicate of source maps or it just add some link to existing source maps and know how to map stack traces with uploaded --url-prefix to that source maps?
  2. In some cases I see that release contain source maps for different --url-prefix, however the command we use to upload source maps using sentry-cli is a common one.
  3. I spotted that services, which has more 20 and less artifacts for upload suffer from that problem, while services with 30+ artifacts do not have that issue.

Maybe you have any thoughts on that behavior?

@Memfisrain
Copy link
Author

It seems, that I found out the issue with debug mode, in logs I spotted that it fetches the artifacts from release:

api/0/projects/panda/appjs-home/releases/f0a1f04d/files/?cursor=&checksum=028993f0a430181870a27e173702060280fe7160&checksum=0329e12d2d36b0c7b044273db2c38279e81b5ca5&checksum=07ab283ff16233ab3fc215bf43d0cc11f49bd49f&checksum=123767323ac50708a077a5165804fb300050fba3&checksum=12ee62dc1f5a10fb442e8df7577fa6406b1b046b&checksum=136f33dc0ce2541a3bee767903a1e9298fec7abb&checksum=1785a870bd336b40ed1a6712fca1133bc6a59df3&checksum=187311de56a575cdb5c91ee88d8cf394d4f52711&checksum=1a067a74243786e97c844ecb7dce6d168050cc49&checksum=1f869d2b0de98537b4b97132022f0c9e8b86aafe&checksum=2b13b68bfa8f267d86a7225e87e6c0462b87c3d0&checksum=2df486d1e6da9516fffc26ae6663f07587e26e1a&checksum=2fbf02e76844b0c1da6953e3a918add6bf47fbd3&checksum=3054d3e0448ab2923f7ffbe306505eab3595c50e&checksum=328fe10d5904de2eddebc50781bc92783685bb46&checksum=34fea3b9af9198b1cb680ebe66a08d761cfca504&checksum=3becc06f087f800cafc5f000baf8ee6d550e51c0&checksum=3fb16fdeab6bcfdb36123a27721ad63531877c6e&checksum=46a6e0cdf1b09ff4332b705bffd4e39bd70facc3&checksum=4893c88455af919a87908b7c378e80b6edbdd594&checksum=52f7c0e12e7918f4adeb2dc48c6be397894db478&checksum=5335927e0d012cf1e21b341fa8ca2c7b632685a1&checksum=54b5d628dd59c43dd1e0870c7289f72539ae3833&checksum=5643b7b69e7763e94292b9d8c447ede44d05b5dd&checksum=62489d7d57be264e6cf5be00a7451bf435454740&checksum=6a306ba186fbcf0eb4b224578aa101bcc02cea43&checksum=6e0df46a7317dce8aa4b3486881eca1a5b4df2da&checksum=6ff785795a703033ec227bbe60c46b790c15e1ef&checksum=814c99504d5bea085a0a4fe83a9e3e16343308cb&checksum=81bf9c2cc036fcca05a0a475867d2b29d6b308fc&checksum=849821dba0ef5b7feac48460e1a9a7501bf698b6&checksum=8529d2c996e5cb1963b55f23c1b6546f0ffe7a69&checksum=85b0844f8b98e1c4129d8ef913f273e6a7d55e1f&checksum=89f9b8e7facc492a6f21ec1bcd35c846c31fbee6&checksum=8ea4125fa571e61629c449f142d8be518294fc8e&checksum=9051786fafe26322b9fcb7825b0ece54292c0fe9&checksum=9081c003591a4293a7be3e1ad203d8a32ef6c97b&checksum=98d64c014e0ffe286d7c4b9745843c2e4bfcd71f&checksum=a04a7c75135a43d401b2cb10b59892bdf5fdfb83&checksum=a1ab231f7f5f4b842765e7bfeb6a7c871b35382e&checksum=a50edf456a4010dc5e95b2a3b5e07273624501c7&checksum=a9ba5489f57c32e403974be9c42bbcd2b42182e9&checksum=ac059da89746906b0904eff50c64ba43c60f12ed&checksum=acd56fbf5953245028b64d8cad6c6d6ace58e3e4&checksum=adc47af88979576e6c3985b2296105b258e5b382&checksum=ae804f3803235518a9d104e30b094e52bc53c4ab&checksum=b27b3e82dc7fb10e713b25162595d24208128f3a&checksum=b50cd30749e450b4c6406bd80c908898896bd9cd&checksum=b7ec7c3aa1e78c7c41e0332e9b0712bc1ce5084f&checksum=b88f9b3edb6ee563edb5d5003b312a08ade613d4&checksum=b9b2f804f669c6018d5cdb88da5e456ea9c5b5e8&checksum=c1880cf246946289a036bec944966a37c4c52cf2&checksum=c36e658d3d690742bd1d53c7392d73d234f8edcd&checksum=c606c866e00b25d921b470a69acc07f00a2516f3&checksum=c957808968e8c8f177c4d0dbe3bc5c5103c0ec53&checksum=d43d2ac70db7cf468d74d9d27b5fda2c95ceb029&checksum=da602b489251bb09e8eea118ff9f18e2b88db522&checksum=e0384d2519f217f56787b5ceac67d6788de2f507&checksum=e809279d1e43e4aabfb486670fcf9dbcb75bf426&checksum=ed459fc4f750ec9650df3976c4faef4f9f96c7ac&checksum=efcfe4773ce0a7185f51d6c85b198d3a9c3cf6f0&checksum=f2f946adc115f4133e5331787e06887c00c7f3ae&checksum=f64135ee15440d1037b079f8ddbd6c62f2f2e2e7&checksum=f929dff8908ef33e82091f76a0d6ee123ff7f31d](https://sentry.infrastructure.pandadoc.com/api/0/projects/pandadoc/appjs-home/releases/f0a1f04d/files/?cursor=&checksum=028993f0a430181870a27e173702060280fe7160&checksum=0329e12d2d36b0c7b044273db2c38279e81b5ca5&checksum=07ab283ff16233ab3fc215bf43d0cc11f49bd49f&checksum=123767323ac50708a077a5165804fb300050fba3&checksum=12ee62dc1f5a10fb442e8df7577fa6406b1b046b&checksum=136f33dc0ce2541a3bee767903a1e9298fec7abb&checksum=1785a870bd336b40ed1a6712fca1133bc6a59df3&checksum=187311de56a575cdb5c91ee88d8cf394d4f52711&checksum=1a067a74243786e97c844ecb7dce6d168050cc49&checksum=1f869d2b0de98537b4b97132022f0c9e8b86aafe&checksum=2b13b68bfa8f267d86a7225e87e6c0462b87c3d0&checksum=2df486d1e6da9516fffc26ae6663f07587e26e1a&checksum=2fbf02e76844b0c1da6953e3a918add6bf47fbd3&checksum=3054d3e0448ab2923f7ffbe306505eab3595c50e&checksum=328fe10d5904de2eddebc50781bc92783685bb46&checksum=34fea3b9af9198b1cb680ebe66a08d761cfca504&checksum=3becc06f087f800cafc5f000baf8ee6d550e51c0&checksum=3fb16fdeab6bcfdb36123a27721ad63531877c6e&checksum=46a6e0cdf1b09ff4332b705bffd4e39bd70facc3&checksum=4893c88455af919a87908b7c378e80b6edbdd594&checksum=52f7c0e12e7918f4adeb2dc48c6be397894db478&checksum=5335927e0d012cf1e21b341fa8ca2c7b632685a1&checksum=54b5d628dd59c43dd1e0870c7289f72539ae3833&checksum=5643b7b69e7763e94292b9d8c447ede44d05b5dd&checksum=62489d7d57be264e6cf5be00a7451bf435454740&checksum=6a306ba186fbcf0eb4b224578aa101bcc02cea43&checksum=6e0df46a7317dce8aa4b3486881eca1a5b4df2da&checksum=6ff785795a703033ec227bbe60c46b790c15e1ef&checksum=814c99504d5bea085a0a4fe83a9e3e16343308cb&checksum=81bf9c2cc036fcca05a0a475867d2b29d6b308fc&checksum=849821dba0ef5b7feac48460e1a9a7501bf698b6&checksum=8529d2c996e5cb1963b55f23c1b6546f0ffe7a69&checksum=85b0844f8b98e1c4129d8ef913f273e6a7d55e1f&checksum=89f9b8e7facc492a6f21ec1bcd35c846c31fbee6&checksum=8ea4125fa571e61629c449f142d8be518294fc8e&checksum=9051786fafe26322b9fcb7825b0ece54292c0fe9&checksum=9081c003591a4293a7be3e1ad203d8a32ef6c97b&checksum=98d64c014e0ffe286d7c4b9745843c2e4bfcd71f&checksum=a04a7c75135a43d401b2cb10b59892bdf5fdfb83&checksum=a1ab231f7f5f4b842765e7bfeb6a7c871b35382e&checksum=a50edf456a4010dc5e95b2a3b5e07273624501c7&checksum=a9ba5489f57c32e403974be9c42bbcd2b42182e9&checksum=ac059da89746906b0904eff50c64ba43c60f12ed&checksum=acd56fbf5953245028b64d8cad6c6d6ace58e3e4&checksum=adc47af88979576e6c3985b2296105b258e5b382&checksum=ae804f3803235518a9d104e30b094e52bc53c4ab&checksum=b27b3e82dc7fb10e713b25162595d24208128f3a&checksum=b50cd30749e450b4c6406bd80c908898896bd9cd&checksum=b7ec7c3aa1e78c7c41e0332e9b0712bc1ce5084f&checksum=b88f9b3edb6ee563edb5d5003b312a08ade613d4&checksum=b9b2f804f669c6018d5cdb88da5e456ea9c5b5e8&checksum=c1880cf246946289a036bec944966a37c4c52cf2&checksum=c36e658d3d690742bd1d53c7392d73d234f8edcd&checksum=c606c866e00b25d921b470a69acc07f00a2516f3&checksum=c957808968e8c8f177c4d0dbe3bc5c5103c0ec53&checksum=d43d2ac70db7cf468d74d9d27b5fda2c95ceb029&checksum=da602b489251bb09e8eea118ff9f18e2b88db522&checksum=e0384d2519f217f56787b5ceac67d6788de2f507&checksum=e809279d1e43e4aabfb486670fcf9dbcb75bf426&checksum=ed459fc4f750ec9650df3976c4faef4f9f96c7ac&checksum=efcfe4773ce0a7185f51d6c85b198d3a9c3cf6f0&checksum=f2f946adc115f4133e5331787e06887c00c7f3ae&checksum=f64135ee15440d1037b079f8ddbd6c62f2f2e2e7&checksum=f929dff8908ef33e82091f76a0d6ee123ff7f31d

As you can see, the url is very long, and maybe because of this this request fails with 502 error and thus services with 30+ artifacts are working good. As I understand, Sentry makes a request to get a files with checksum for comparison and in this case comparison is failed and it uploads artifacts again. On the other side, when we have a small amount of artifacts, that request is not so long and it handles correctly, returning the list of existing artifacts. Thus it compares to the uploading source maps and skip it.

@Memfisrain
Copy link
Author

And one more question: we use @sentry/cli npm package and call that Sentry stuff in typescript code. I actually do not see possibility to use this --no-dedupe flag in that module. You can check typings here: https://github.com/getsentry/sentry-cli/blob/master/js/index.d.ts#LL66C4-L66C4

@kamilogorek
Copy link
Contributor

Ah, you're right, updated in the PR.

As you can see, the url is very long, and maybe because of this this request fails with 502 error and thus services with 30+ artifacts are working good. As I understand, Sentry makes a request to get a files with checksum for comparison and in this case comparison is failed and it uploads artifacts again.

You're totally right. We've been bitten by this few times already, and we want to change this check in the future to not use query params, but rather pass this data in the body instead.

How that source maps are stored on the server and in case I uploaded the same source maps but use different --url-prefix does it upload and store a duplicate of source maps or it just add some link to existing source maps and know how to map stack traces with uploaded --url-prefix to that source maps?

This is changing now, as we're moving to debugIds instead of relying on fragile urls; You can read about them here - https://sentry.engineering/blog/the-case-for-debug-ids

@Memfisrain
Copy link
Author

Thank you! After you merge the PR and create a new release I will give it a shot and hopefully the issue will be resolved

@Memfisrain
Copy link
Author

Seems working, thanks a lot for quick reaction!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants