Feedback on the new 'Flags' tab #27
Comments
|
There should be an easy way to delete flags from the flags tab. All of our flags are carry forward: flag_management:
default_rules:
carryforward: trueWhich means we have lots of dangling flags left over from when we were configuring codecov for the first time. We experimented with different naming conventions and such. There's no way to clear these flags out, and I'm worried they will contribute to weird coverage numbers. |
Agree +1 |
|
We really appreciate the feedback! We've heard the need for clearing out older flags and are looking at it as work to be done in the next few months. We also have a couple other improvements to the flags tab that will allow you to search and select only a few flags you want to see at a time. |
|
Hi there, We have a few pieces of feedback that I think would really improve the flags page:
Our use case is that we want the ability to track individual team coverage over time (where each team is a flag) on our monorepo. Right now because of the way that the flags report is averaged over time, across all PRs this severely limits the usefulness of it. |
|
After enabling the feature, the status stays at "Pulling historical data" now for roughly half an hour - although we enabled it right after the first scan on the default branch. What could be the reason? Let me know if you need more info. |
|
@FelixS90 our historical data pull feature relies on an asynchronous task queue to function. This is essential since, in many cases, users have a a large amount of historical coverage data that we need to pull, aggregate, analyze, and store in a time series friendly format. With this in mind, there are two reasons a historical data pull can take awhile:
We generally try to autoscale processing for that queue to keep queue length under control, but it could have just been a particularly busy time in your case. Apologies for the delay, but is the feature working for that particular repository now? |
|
I get a consistent "There was a problem getting flags data" after a considerable wait time. JS console output: |
|
I don't understand the flags. In the flag section it's stated 5.5% But looking a the normal page and the folder that is tagged, that folder is 17.5% My idea was to have each package have it's own coverage but they don't match in the flag section. |
|
I am not seeing data for my configured flags 🤔 |
|
I also see "There was a problem getting flags data" - it appears to work once every 10-15 attempts, but only after a lengthy delay. From browser console: |
|
Hi all, we are actively working on the issues causing the error states, when you encounter those, changing the time scale can oftentimes unblock you. @RobertBrunhage I'd love to dig in further with you on your particular issue, can you give me a specific repo/flag (if it's public) so I can hopefully get you an answer? |
|
Is this a known issue where some of the flags are empty?
|
|
@Wayonb flags stick around once they have been created at any point in time, for example 'sdk-ptyhon' was likely a flag at one time, which was corrected to 'sdk-python'. In the case of the empties you see in your flags tab, is it possible each of those flags was uploaded to once and then never again? The lack of data makes me wonder if they were even uploaded to with a blank coverage report? It's hard to say exactly without more info, but those are the possible avenues I'm aware of that might cause this scenario. |
|
@aj-codecov Thanks. I guess the only way to remove those is to reset all the data? |
|
From #27 (comment) :
I thought so as well. Is that true for flags feature? My case is Xabaril/AspNetCore.Diagnostics.HealthChecks repo https://app.codecov.io/github/Xabaril/AspNetCore.Diagnostics.HealthChecks/flags with ~40 projects and I would like to track coverage independently for each project (package). Can I archive that by flags feature? I planned to add |
|
I've followed the instructions, but all I get is
Clicking the link to show the reported uploads shows all the coverage data, so no idea what is going on. I've had to disable this entirely now since it no longer reports any coverage at all when I use flags, which is a bit of a pain. There is nothing to actually indicate what the issue is though, so for now I won't be able to make use of this, which is unfortunate as this feature would be really useful in my project for separating out integration and unit test coverage. If there is a way to feed back errors more clearly on these pages, it would be greatly appreciated and of great use in the future. |
|
@sungam3r I believe you've got the right idea, only thing I'd say is if you're not uploading all coverage every time you commit, you're going to want to use Carryforward flags as described here: https://docs.codecov.com/docs/carryforward-flags |
|
@ascopes In your first screenshot I can see you've attempted to tie two flags to one upload, Flags must be associated with only one upload to work properly, so you would need to do an upload of coverage for integration tests and an upload of coverage for unit tests where each is flagged with their corresponding label further reading here: https://docs.codecov.com/docs/flags#one-to-one-relationship-of-flags-to-uploads |
|
@aj-codecov I have configured Also I set individual but after the latest upload Xabaril/AspNetCore.Diagnostics.HealthChecks#1616 I still see only coverage from only touched files and overall coverage dropped again after full report upload. |
|
@aj-codecov Hi, thanks for the response
I was under the impression that the feature flags would handle this automatically since they consume specific paths related to specific flags. Is this not the case? flag_management:
default_rules:
carryforward: true
statuses:
- type: project
target: auto
threshold: 1%
- type: patch
target: 90%
individual_flags:
# Unit testing coverage
- name: unit
carryforward: true
paths:
- '**/target/site/jacoco/unit/jacoco*.xml'
statuses:
- type: project
target: auto
threshold: 65%
- type: patch
target: 90%
# Integration testing coverage
- name: integration
carryforward: true
paths:
- '**/target/site/jacoco/int/jacoco*.xml'
statuses:
- type: project
target: auto
threshold: 0%
- type: patch
target: 0% |
|
My flags page is not loading the coverages:
Any idea what's happening? LogsThe connection to https://clientstream.launchdarkly.com/eval/60f9b3cf32beb9244b10f3d1/<SOME_LONG_TOKEN> was interrupted while the page was loading. main.ad642d24.js:2:550053
[LaunchDarkly] Error on stream connection: {"isTrusted":true}, will continue retrying every 1000 milliseconds. main.ad642d24.js:2:407351
Sentry Logger [error]: Error while triggering instrumentation handler.
Type: fetch
Name: <anonymous>
Error: TypeError: o is undefined
Et https://app.codecov.io/static/js/main.ad642d24.js:2
en https://app.codecov.io/static/js/main.ad642d24.js:2
m https://app.codecov.io/static/js/main.ad642d24.js:2
h https://app.codecov.io/static/js/main.ad642d24.js:2
main.ad642d24.js:2:414414
99204/s/</t[n]/< https://app.codecov.io/static/js/main.ad642d24.js:2
u https://app.codecov.io/static/js/main.ad642d24.js:2
n https://app.codecov.io/static/js/main.ad642d24.js:2
m https://app.codecov.io/static/js/main.ad642d24.js:2
h https://app.codecov.io/static/js/main.ad642d24.js:2
Sentry Logger [log]: No outcomes to send main.ad642d24.js:2:414414
Some cookies are misusing the recommended “SameSite“ attribute 7
Segment snippet included twice. flags line 512 > injectedScript:1:126
Sentry Logger [log]: Integration installed: InboundFilters main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: FunctionToString main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: TryCatch main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: Breadcrumbs main.ad642d24.js:2:414414
Sentry Logger [log]: Global Handler attached: onerror main.ad642d24.js:2:414414
Sentry Logger [log]: Global Handler attached: onunhandledrejection main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: GlobalHandlers main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: LinkedErrors main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: Dedupe main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: HttpContext main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting pageload transaction on scope main.ad642d24.js:2:414414
Sentry Logger [log]: Setting idle transaction on scope. Span ID: 89231477b0e5b4a9 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because it's not included in the random sample (sampling rate = 0.2) main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: BrowserTracing main.ad642d24.js:2:414414
Sentry Logger [log]: Integration installed: Replay main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'ui.react.mount' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'ui.react.mount' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 2 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 2 main.ad642d24.js:2:414414
Loading failed for the <script> with source “https://static.ads-twitter.com/uwt.js”. flags:1:1
Sentry Logger [log]: [Replay] Using compression worker main.ad642d24.js:2:414414
Ignoring unsupported entryTypes: element. main.ad642d24.js:2:153138
Ignoring unsupported entryTypes: largest-contentful-paint. main.ad642d24.js:2:153138
Ignoring unsupported entryTypes: layout-shift. main.ad642d24.js:2:153138
Ignoring unsupported entryTypes: longtask. main.ad642d24.js:2:153138
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). 4 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://clientstream.launchdarkly.com/eval/60f9b3cf32beb9244b10f3d1/<SOME_LONG_TOKEN>. (Reason: CORS request did not succeed). Status code: (null).
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Request to access cookie or storage on “https://googleads.g.doubleclick.net/pagead/viewthroughconversion/781097871/?random=1674002832456&cv=11&fst=1674002832456&bg=ffffff&guid=ON&async=1>m=2wg1a1&u_w=1440&u_h=900&hn=www.googleadservices.com&frm=0&url=https%3A%2F%2Fapp.codecov.io%2Fgh%2Fvigenere23%2Fdisma%2Fflags&tiba=Codecov&rdp=1&auid=1383083016.1674002832&rfmt=3&fmt=4” was blocked because it came from a tracker and content blocking is enabled.
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] No active IdleTransaction main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because its trace was not chosen to be sampled. main.ad642d24.js:2:414414
Sentry Logger [log]: Adding outcome: "sample_rate:transaction" main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (89231477b0e5b4a9). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] No active IdleTransaction main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting navigation transaction on scope main.ad642d24.js:2:414414
Sentry Logger [log]: Setting idle transaction on scope. Span ID: 80df6c1167066ba2 main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because it's not included in the random sample (sampling rate = 0.2) main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/gh/vigenere23/disma/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/gh/vigenere23/disma/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Starting 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
XHRPOSThttps://api.feedback.us.pendo.io/widget/pendo_ping
[HTTP/2 400 Bad Request 145ms]
Sentry Logger [log]: [Tracing] Finishing 'http.client' span on transaction '/:provider/:owner/:repo/flags' (80df6c1167066ba2). main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] No active IdleTransaction main.ad642d24.js:2:414414
Sentry Logger [log]: [Tracing] Discarding transaction because its trace was not chosen to be sampled. main.ad642d24.js:2:414414
Sentry Logger [log]: Adding outcome: "sample_rate:transaction" main.ad642d24.js:2:414414 |
As an update, I've realized that all my recent uploads have failed, with the message |
|
Another update : after fixing the coverage upload, it does load the information now, thanks! |
|
This feature is pretty neat but I have 2 comments on it:
|
|
Dunno if I am using Flags the right way, but I felt they might provide a good introspection for which parts of my code are covered with which tests (unit, integration, system). Unfortunately, the UI only gives the total percentage of coverage, without any detalization per file. What I would like to see is something like this:
|
|
It would be very helpful to display the latest report upload time on the Flags page! |
|
I like the feature, but I'm also having issues with weird coverage showing, for example:
P.S. repo is open-source. |
+1 |
|
I would really like to have:
|
|
We would like the ability to delete a flag we have retired so we do not keep getting it on comments with carry forward at the same level. |
|
Hi, |
Bump for this suggestion. I have |
|
I think it's worth creating a feature request for that. I'll do soon. |
|
@iwishiwasaneagle --> When did you take this screenshot and are you using Codecov Cloud or self-hosted? We now have a flag selector live on all repos, see below: https://app.codecov.io/gh/codecov/worker?flags%5B0%5D=integration Or you can post the repo here and I can take a look as to why this is not showing up for you. I don't think it's feature flagged to only some users (@aj-codecov can confirm). 
|
|
@iwishiwasaneagle Jerrod is correct - if you're using flags and using Codecov cloud, you should see the selector and be able to do exactly what you hope - the only quirk might be if you haven't uploaded flags data on your most recent commit to the chosen branch, then we wouldn't have flags data to filter for that commit. Feel free to drop a repo you're having trouble with though and we can dig in for you! |
|
Apologies all, I never saw the homepage flag selector. To me it was intuitive that browsing the files for each flag is something done from the flags submenu.
…-------- Original Message --------
On 20 Nov 2023, 16:56, aj-codecov wrote:
***@***.***(https://github.com/iwishiwasaneagle) Jerrod is correct - if you're using flags and using Codecov cloud, you should see the selector and be able to do exactly what you hope - the only quirk might be if you haven't uploaded flags data on your most recent commit to the chosen branch, then we wouldn't have flags data to filter for that commit. Feel free to drop a repo you're having trouble with though and we can dig in for you!
—
Reply to this email directly, [view it on GitHub](#27 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AC6PGNNZTM5PMF5MNF4Y4Z3YFODTFAVCNFSM56XNDTLKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBRHE2DKNJWGUZA).
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@iwishiwasaneagle it's really no problem -- users should be able to know where to go intuitively, and if they can't, it's worth us knowing. UPDATE: Ignore the below, the screenshot was from September. |
|
@jerrodcodecov it was the reply to my post made few months ago. People liked my post so I suppose there were more people who didn't have this ability. There wasn't such dropdown presented and therefore I asked about it here. |
|
Got it, I totally misread the the thread. Thanks for sorting me out @skynetigor and you can ignore me @iwishiwasaneagle . This feature was just released this month on a rolling basis and will be part of this month's product newsletter and blog. https://about.codecov.io/resources/ In the meantime, you can always see all changes in the last 30 days here, but more difficult to read/parse: https://github.com/codecov/gazebo/releases Jerrod |
|
Thanks @skynetigor for clearing that up! Here's a screenshot of what I mean @jerrodcodecov to further clear this up. Even just a tool-tip on hover would be useful.
|
|
agree with @iwishiwasaneagle |
|
@iwishiwasaneagle @skynetigor yep, it's just a really smart and relatively cheap feature idea. We will make it so. We can probably make it more elegant in the future, but as fastest implementation we will probably have it drop you directly onto the files explorer and coverage chart filtered by flag already. Like this URL, for example: https://app.codecov.io/gh/codecov/worker?flags%5B0%5D=integration |
|
Update: feature is now in pipeline. codecov/engineering-team#847 h/t @aj-codecov
|
When I use this, it filters the files to only the ones in that flag, but the coverage percentage still shows the results from all flags. If I open a file, then it shows me the percentage for that flag, which is rather confusing and making it difficult to analyse total coverage of a flag.
|
|
I am facing the same issue as @Dreamsorcerer . Was so excited to use the flags feature. But now it is proving difficult to enforce patch level checks on a particular flag. As team is finding hard to figure out which lines are covered by what test. |
This is codecov/engineering-team#1269. @codecov/design , @codecovdesign is this work being handled in a different issue? |
|
@aj-codecov we started using the flags. We've encountered an issue in which flags with dash are being cut of, for example
|
|
After upgrading to version 5.0.0 the issue was resolved. |
|
@shivang-certify we reviewed the issue you described, where the patch was reporting as 100% and the flags reported differently. This appears to be a bug and we created a bug issue to investigate. Thank you for raising and will add a note to follow up on the results. |
and others
Thanks for dropping by! 👋
We've recently released a whole new area of the app focused on flags. If you're not seeing it yet, keep your eyes peeled as we're slowly rolling it out to our users.
We greatly appreciate your time and thoughts - looking forward to hearing from you ❤
Codecov team
This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please see our support page.
The text was updated successfully, but these errors were encountered: