Proposal: rework "groups" into "tags"

[kimeberz]

Today's groups feature was added a while back without a whole lot of thought.

There are a few quirks with its behavior:

• If your pipeline configures groups, all jobs must be present in the groups section, otherwise they will not show up in the UI at all. There is no feedback in the UI for this, and there's no way to view all jobs in the pipeline, so your job becomes a sleeper cell.
• Configuring groups means maintaining a list of your jobs in one place, and then configuring the jobs somewhere else. This means more hopping around than normal; jobs and resources are otherwise self-contained.
• It's not at all discoverable that you can view multiple groups by shift-clicking them. (WHAAAT? YOU CAN DO THAT? yis.)
• The UI doesn't handle either having many groups or having long-named groups. We should probably handle it more elegantly.
• We show the first group on the front page, which is a little hard to discover (though we do document it), and pretty inflexible, since sometimes you may want multiple groups viewed by default.

A Modest Proposal:

1. Reimagine groups as "filtering jobs by tags".
2. Remove groups from pipeline config (well, keep it for backwards-compatibility).
3. Each job should now have a tags field, which lists the tag names for the job.
4. The pipeline can declare a default_tags field, showing what tags the pipeline will be filtered to by default.
5. This filter can be cleared to show all jobs, including jobs with no tags.
6. The UI would be intuitive and not suck.

• @kimeberz & @vito

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #140235293 Proposal: rework "groups" into "tags"
• #121271557 Proposal: Rework "groups" into "tags"

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[cjcjameson]

Hi there! Bumping this because we have multiple teams here that have stubbed their toes on #198 this week. I agree with @kimeberz 's general proposal but just reaffirming that the original problem still exists, and if there's a way to improve groups without rearchitecting, it would be valuable.

^ @ashwinstar @chrishajas

[cjcjameson]

Bumping again because we just rehashed this UX thing again. Use case: https://github.com/greenplum-db/gpdb/pull/2851 Tagging @pivotal-mike @kmacoskey @divyabhargov @doty-pivotal @jingyimei

[rkoval]

It didn't seem like it was explicitly discussed, so I'll ask: does this proposal include consideration of grouping by tags across all pipelines (vs. just the tags in your current pipeline config, similar to how groups behave now)?

I'm not sure how much extra complexity this would add, but it would be immensely helpful to enable global tags because you would be able to group different pipelines together without having to cram different job configs in the same file. Our team would then be able to consolidate a lot of pipeline config into reusable templates that would be deployed as separate pipelines. Right now, we have a looot of duplicate of config simply to get the groupings in concourse. It's looking like we'll abandon using groups altogether as a result for the time being until something like this proposal gets implemented. (I could be missing something with how to utilize groups though! I'm still pretty new to Concourse 😅 )

[EugenMayer]

If you implement tags, which would be very much needed, there should be a consideration of adding a search like query ( very simple ) to find jobs to display and that query should be able to be added as menu item. ( like a saved JQL expression )

The point is, you will need different "views" on different the same jobs your current "hat" ( release hat / staging hat / backtracing hat). So you will use more tags and try to create different sets in the menu to help out.

In the long term, its obvious that structured classification is needed. Creating pipelines for actual projects, which then can be part of "teams". Probably also something like an "environment". This would make it possible to make proper generic UI to help people out without wrapping their head around how to utilize tags to build something very similar ( again and again, for every single concourse instance )

[jama22]

@rkoval @EugenMayer its something we've been thinking about a lot but we have so many things in flight that we might not get to this in a few more months. I'll dump out a few notes and conversations that I've had with @vito here:

• Groups has definitely run their course with the current UI. While its useful for a handful of people, more folks are looking for the tagging and search system described in the original comment
• Our first experiment / foray into what "search" means in Concourse is with the Dashboard UI [Search] Search by pipeline status #1703 [Search] Search by team name #1684 [Search] Search by pipeline name #1682 . We plan to extend those concepts into the pipeline view with the ability to search on jobs, names, and tags (if we actually get around to converting groups to tags!)
• We've also hit the limit on the design metaphors and efficiencies of the current UI. SVG is really hurting us and we want to move towards something simpler, snappier, and more responsive.

That said, I think @rkoval 's proposal is very interesting to consider for tags. Implementing global search across the context of your team could be very powerful, especially if Concourse is being used for complex deployment pipelines with separation of duties. Perhaps we could consider a global: keyword on search to tell Concourse to look for tags outside of the current pipeline context and include all pipelines within your current team's realm of concern.

[EugenMayer]

Well yeah the current UI is blocked by technology, thats what i have seen already.

You might want to switch to something more fitting for CI system, ng4/react with something like https://github.com/akveo/ngx-admin to kick of very easy, with probably https://www.jointjs.com/ for the pipelines (json backend based, interesting for automation of the graphs) using things like http://resources.jointjs.com/tutorial/multiple-links-between-elements. For the graphs thought there are probably simpler ones as you only want to define a graph by code, no editing involved. We are actually offering/developing a Flowchart editor, which of course has SVG in its editing part, so i def. know what you mean that SVG is hurting. Those JS layers help a lot though.

The current UI is probably a conceptual issue and maybe there is little sense to patch it, maybe replace it rather. Considering how little the UI actually does and offers, this could probably be not as much work as expected - also the UI could be a dual-head thing, so start with a new UI, since it is decoupled AFAICs you can drive the second one in parallel for quiet a time probably, not forcing yourself into a lot for trouble.

But i rather not get involved here, not in need for another beating. Thanks for considering this topic in general.

[jama22]

Yes, that's exactly what we're doing @EugenMayer , we've decoupled our UI work with our other backend work! Thanks for the suggested libraries...we'll probably start investigating those as we roll out our new designs in the beta route. But to be honest, we've always hated the idea of layering more jS to solve our problems; makes things hard to test reliably.

You are also correct sir, we're rethinking the UI considerably. That's why we have our not-so-secret beta/ route to test new ideas and UI concepts 😏

[EugenMayer]

I was not aware of the beta at all, can i see / test it, or is it internal use only?

[jama22]

@EugenMayer beta was added in 3.6.0. Apologies, I think i gave you the wrong route, try beta/dashboard/ and you should see the new dashboard with the search capabilities I mentioned. Then click on an item to see a completely broken version of pipelines :)

[stale]

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.