Discovery: configuration of some or all features (Codecov Pre-Release onboarding)

[codecovdesign]

Problem to Solve

Context: given upcoming new feature development, a problem on the horizon is addressing the onboarding experience for repositories with varied needs in terms of feature configuration. The separate features with varied configuration:

1. Coverage (core product)
2. Bundle Analysis
3. Test Result Ingestion
4. ATS

Users might be interested in configuring one, multiple, or all of these features, either at the onset or progressively over time. This presents a challenge as we need to accommodate varying user needs and scenarios within the onboarding experience.

Discovery

• Ideation i

  • related feedback

• Ideation ii

  • related feedback

initial issue details

Questions

• Rough configuration specifics: a rough outline of the key configuration requirements for each feature. For instance, detailing that a failed test should be inputted in codecov.yaml for Test Result Ingestion.
• Do these features all live together OR do we want to start separate them out into a more modular display.
• Consider other status: slack app, CLI etc, could these be marketed in UI (status on/off)
  • other scenario where ATS shows that it's configured but doesn't have any data: ATS Dashboard view for repos not using ATS (leads to growth in usage) #811

WIP: initial iteration

WIP: longer term iteration

Tasks

Beta Give feedback

1. Design: persona, JTBD, and stakeholder map #917
   +0
   
2. Not yet configured status updated #977
   Dev-Ready epic +2

Tasks

Beta Give feedback

1. Technical Planning for pre-release UI changes #1338
   investigation +1
   
2. Commit and Pulls Polish #1497
   0 of 1
   Dev-Ready Polish +2
   
3. Feature in-app marketing and configuration status #1509
   in discovery +1
   
4. Information architecture: better structuring coverage and subcategories #1537
   3 of 4
   in discovery +1
   
5. Update nav, header, and related elements #293
   0 of 21
   Dev-Ready epic +2
   

[codecovdesign]

Issue Summary

This is exploring a rough concept for a pre-release experience, considering current knowledge and the assumption of ongoing investment in all features. This is an exploratory phase, not a concrete proposal, to outline a potential multi-product direction, contingent on new product adoption and MVP development. Key considerations:

• GitHub App Requirement: The necessity for GitHub app installation continues, meaning the current signup process facilitating this will remain essential.
• Feature Integration: Each feature demands specific repo and/or CI integrations, implying no universal one-click setup due to distinct onboarding requirements like tokens and CI configurations. However, opportunities for unified configuration should be leveraged to enhance cross-feature usability. Ideas include a shared YAML or a configuration helper with ready-to-use templates (related issue).
• Shared UX Touchpoints: Despite separate configuration UX for each feature, common elements fitting the "pre-release" phase can be integrated, such as shared reporting in commits, pull requests, and PR comments.
• Converging Feature Themes: Finding common themes across features can help unify them under broader categories (e.g., "Testing" could encompass coverage, test ingestion, and auto-test selection) while maintaining distinct offerings.

The overarching goal is to map out an information architecture that lays the groundwork for a cohesive onboarding experience across multiple products. Based on the above, it could look something like this:

The UI could translate then to:

Interactions demo:

architecture_interaction.mov

With multiple products used, an area of convergence in the UX is under pre-release reporting, here we see the PR comment aggregates the reports into one:

A shorter summary view may be preferred, based on previous feedback, here is example with warnings and ability to expand for further details:

Same, but in the scenario where all reports are ✅

In-app experience could mirror the converged UX as all features reports share the git commit/PR workflows:

In closing, it's worthwhile to note that this exploration is contingent on the outcomes, progress, and specific configuration needs of each product. In the shorter term, weighing critical questions and tradeoffs focus on i) the success of new products like bundle analysis, ii) the implications of configuration/ui/ux/growth/adotpion for testing ingestion, and iii) the potential of experiments such as Automated Test Selection (ATS) and AI Review.

[codecovdesign]

applications sync 2/22 review:

• nick: today bundles is separate in pr comment; rohit: would be noisy to have multiple, helpful to aggregate
• group: how does integration look potentially with the existing sentry comment? tbd
• general note: would need quite a effort to make everything response (TODO investigation IF needed)
• aj: also nice that the pattern matches sentry navigation
• rohit: could be interesting to user tests perception of logo/branding
  • adalene: google has multiple products in one, looks similar but different, could be separate apps

@eliatcodecov review 2/23

• another area to consider here is what is the minimal next step and/or what would be the cause for the next step to be proposed (such as X happening to certain feature)

@rohan-at-sentry mention in slack thread:

I agree that we should wait until we're closer to Q2 until we start fleshing this out more...

@drazisil-codecov @vlad-ko

• Joe: I like this, but would like to see the PR comment all come together.
• Joe: Configuration as multiselection config to repos would be great
• Vlad: the pattern left nave, like sentry makes sense

[codecovdesign]

Follow-up review

Following up on the feedback and deepening our exploration in another area shared amongst potential products is the git workflow—repos, commits, and pulls. Our previous discovery showed potential for a unified reporting approach in the app UI pre-release, particularly for the aggregated PR comment experience, raising some questions:

• Will our workflow modeling continue to adhere to git's workflow, or might we consider a unique organization structure (example, like Sentry, where a custom org is created vs inherited from git)?
  • Will proceed assuming it's the former: git workflow
• Is displaying organization-level data in the pipeline? More details here.
  • Not critical, but does influence the architecture
• Regarding repo workflows, this an an area for consideration of our approach. One option is to display all repositories as they are currently, where leveraging pre-release aggregated data could benefit from implementing a pinning pattern. This pattern could also enhance the left navigation by keeping pinned repositories easily accessible. Alternatively, we could consider a workflow where repositories are actively selected for configuration and displayed prominently once one or more features are set up. Both approaches have their advantages and drawbacks (nor are they mutually exclusive), and while it may be too early for detailed exploration, they're important to acknowledge for future consideration.

pre-release_2.mov

Consideration: pre-release is the way?

The path to a solid proposal is challenged by the lack of product validation, which makes decisions and tradeoffs about navigation overhaul investment return unclear. However, the aim of moving beyond coverage to a more wide reaching "pre-release" tool is something that is confirmed and pursuing in any case. This leads to some questions: should we adopt a more accommodating navigation pattern moving forward - no matter what the success is with an experiment like bundle? Consider, if it's not bundle it may be the next experimental product (test ingestion, AI review, ATS, etc) that could leverage this architecture 🤔

The prototype includes a configuration page, aiding users in understanding feature setups across repos and encouraging the discovery of new features for adoption.

Here's a demo illustrating the navigation from organization and repo selection to configuration (onboarding), commits, and coverage, showing high-level data integration across products.

Repo configuration new user	Repo configuration w/ configured features	Repo OVERVIEW w/ pre-release data summary

pre-release_3.mov

Moving forward

In summary, the two key questions here are:

1. Will our workflow modeling continue to adhere to git's workflow, or might we consider a unique organization structure (example, like Sentry, where a custom org is created vs inherited from git)?
   • Will proceed assuming it's the former: git workflow
2. Should we adopt a more accommodating navigation pattern moving forward - no matter what the success is with an experiment like bundle? Consider, if it's not bundle it may be the next experimental product (test ingestion, AI review, ATS, etc) that could leverage this architecture

Let's say the answers are 1) yes, proceed with git workflow, 2) yes, pre-release is the way; then this builds confidence in proposing a scalable UI solution. A minimal iteration could look like this:

Screen.Recording.2024-02-29.at.7.15.26.AM.mov

If we'd like to move this forward, next steps would be:

• Engineering investigation for review / breakdown
• Design deeper diver prototyping UI elements / breakdown

[codecovdesign]

2/28, @katia-sentry

• the git workflow makes sense; question to consider scenario where it doesn't, but seems like a sensible constraint to move forward with
  • pre-release does feel confirmed, so it does seem that the more accommodating nav could help as we will keep running into the same non-scalable issues brought up by bundle
  • helps to see the configuration for discovery, also another is the browser extension and product offerings could be more visible. this would help bring light to the features.

2/28 @rohan-at-sentry

• Everything is tied to a notion of a commit which is a tying and fundamental point. Org is collections of repos, then pr group commits, and then therefore we have a commit based workflow. What we don’t have is multiple orgs per customer, fair to assume one customer per org since this is plan level. So this tracks.
  • It becomes a bit interesting at the settings level
• Configuration page: one key problem that we start to run into is things outside of coverage are not discoverable. One of the benefits of having a structured way of showing is it gives you discoverability.
  • This would be powerful as a first step, it’s not sufficient
  • Improving navigation also feels scalable and also benefits a world even if it’s just coverage and another product
  • Benefits here are the growth and discovery of new features; and it may be low effort enough to build for the future.

2/28 @Adal3n3

• (from slack thread) I did some exploration on the failed test in-app onboarding and dashboards exploration. I had a walk thru section with
  today, and the big question is how would the customers want to compare/view their data? Do they want to click thru the repo to find a file and view the metrics or would they want to compare 2+ files/repos metrics.

2/28 @aj-codecov

• One area that may be something is a greater integration into Sentry. Unsure, but it would be surprising if we weren’t asked to have something. Could be a sentry org inheriting a Codecov org. There might be a way to map to sentry organization somehow, not much other than thinking it’s a safe bet.
  • Hard to imagine pre-release that doesn’t map to the git workflow
  • Having everything in a respective commit and/or PR review is valuable, though problems to solve are if we are having too much into one. Is the PR comment a mess? We’ll need to be careful as we expand in this regard
• Configuration
  • Could we force configuration of yaml to include flags
    • Does flags work in global yaml
  • Is there something we could do by default that carries over across repos
  • Appreciate the thought process of the discoverability and clear path for new features and releasing it into the product

2/29 @drazisil-codecov @vlad-ko

• Bulk configuration should be made possible, possibly through a global YAML configuration, something to consider for later.
• Regarding the repo guide page, some customers might skip it and enable multiple repos with a global upload token.
• Version 1 isn’t necessarily ready in terms of org-level configuration, but considering how to configure across org repos is very important.
  • Vlad: the current bundle setup doesn't make sense, but moving to an easier navigation aligns with Sentry and makes sense.
  • Joe: the left nav is a familiar pattern and helps users. It’s much easier and what people expect. I'd like to see a standard flow where categories are on the left side and breadcrumbs mirror this. Once we transition to this, it will be easier to integrate new features due to nesting.
  • Joe: navigation feels natural; however, Vlad is unsure if displaying all data at the commit level is ideal as it could become noisy with multiple data points.
  • Joe suggests having a summary view is nice, but the commit-PR landing could be configurable to show different columns.
• The default is to aggregate data at the PR/commit level page, with PR expand/collapse control and configurable data at the PR/commit lists level.
  • Consider scenarios where customers may want only one data point, such as an overview of flags over time.
• Summary: The next iteration to the left navigation is a significant improvement for the majority of customers. This foundation makes it easy to add new features without reinventing the site, reducing clutter with a concise tree of categories.
• Configuration: Joe notes it helps users as it's clean and easy to understand.

3/1 @eliatcodecov

• agree with information architecture and q/a that got us here. Unsure of other flows, such as pr comment to pulls page, and other unknown/unintended consequences and mapping that out will help us make this concrete. Would be great to get in the weeds to map everything out
• We could consider this for Q2 work; while in tandem we see how bundles pans outs
• Let’s move forward with investigation for more information about costs and evalutate whether to proceed then

[codecovdesign]

Next steps

Following our discovery and received feedback, here are the next steps:

• Engineering investigation: assess technical requirements and potential costs.
• Design prototyping: refine UI elements, workflows, and the implications.
• Team review: look at the findings and iteration proposal and decide path forward.

---

Under review:

Information architecture

Flow: PR comment to app

PR_toapp.mov

Flow: org > org selection > settings > repo selection > coverage > bundle > commits > individual commit > coverage reports expanded > minimize left-nav

app_workflow.mov

Additionally, let's look at following concept to report back to user how/what a repo is configured with status:

Repo configuration new user	Repo configuration w/ configured features

[codecovdesign]

sync with @nicholas-codecov @eliatcodecov @aj-codecov @nicholas-codecov @katia-sentry @spalmurray-codecov on 3/22

re left nav concept:

• commit page could be jammed packed
  • we can trim the commit/pr list to the essentials
• eng: @nicholas-codecov WIP a general rough draft of cost / tradeoffs related to the left nav concept

other:

• discovery: explore top level style, w/org/repo selector (@codecovdesign ToDo create discovery issue)
• discovery: explore repo configuration, per import workflow (@codecovdesign ToDo create discovery issue)
• in tandem move with smaller polish iterations such as: Radio button selector #1439 / Align repo guide styling #1437

notes for future left nav design consideration from async:

• fluid layout
  • repo pag sunburst / line graph swap
• on left nav ability to return to org > home (other than breadcrumb)

[nicholas-codecov]

figma

[codecovdesign]

Discovery closing

Navigation

Here are different paths forward:

• Left navigation
  • Steps: navigation overhaul ⚠️ See more from Nick's eng report
• Existing top-nav iteration
  • Steps: 1) nav/header cleanup, 2) correct info-architecture

Tradeoffs for left navigation include:

• Refactoring to fluid design
• Increased overhead for mobile responsiveness
• Codebase restructuring
• User adaptation to new UX

While left navigation offers benefits; the challenges, costs, and updates required are notable tradeoffs. On the other hand, the top-nav approach, which utilizes our existing structure is less costly. Below, is a path that updates our existing information architecture correctly and includes a new toggle component, seen on the repo guid page:

update_secondary_IA.mov

The main tradeoff here is relocating components/flags below coverage, aligning with the correct information architecture. This direction seems scalable for future feature updates while reduces nav/header clutter.

Discoverability

We are aiming to improve pre-release feature visibility and management (and plan differentiation + upgrade opportunity); notably, testing shows users often go from docs > repo > settings for configurations. This issue proposes renaming 'settings' to 'configurationand addingrepo management` section to help user manage repo and feature discoverability. As the section evolves it could support future improvements such as a YAML builder or configuration via PR, clarifying feature availability per plan.

config-slack.mov

Another is looking at sign-up onboarding scenarios to help guide, per user intention: #1340

Shared Reporting

Our current solution, commit/pr UI, supports multiple reports and looks to be a sustainable pattern; however, iterations in polish and interaction (such as direct linking from PR to expanded report, per comment) support UI/feature clutter mitgation.

An in-progress example of polish toward reducing clutter is the cleanup of commit/pulls list, as shown here: Commit/pulls list cleanup

problem: clutter on commit/pr list	update: clear clutter and focus most important data