Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic: spatial resource flows #1707

Open
vito opened this Issue Oct 10, 2017 · 30 comments

Comments

@vito
Copy link
Member

vito commented Oct 10, 2017

Feature Request

What challenge are you facing?

Concourse makes it very easy to model changes over time. Resources do this for you; you point them at a given source of truth, and it'll let you know everything that happened.

Some workflows, however, can't just be modeled as change over time. Multiple branches of a repo, or pull requests to a repository, are over space, not time. They are parallel pipelines, which today are hard to manage, and impossible to correlate (you cannot fan-in).

Build matrixes are another example of wanting to run over many spaces (i.e. versions of a product). This can be done today, within a pipeline, but results in a massive pipeline with every combination explicitly configured.

A Modest Proposal

This issue ties together #1172, #1176, and introduces the ability to have arbitrary build matrixes within one pipeline, which should dramatically improve the experience for people testing many variations/combinations.

Resources would have an additional interface to discover spaces. What these are depends on the resource. For the git resource, it would return each branch. For a resource tracking pull requests, it would return each PR number. For a resource tracking a semver'd product, it'd emit one space for each supported version line, at each granularity (i.e. 2, 2.1, 2.0, 1, 1.9, 1.8, ...).

All versions would then be for a respective space, and Concourse would detect all versions across all spaces.

Jobs could then specify which spaces their get and put steps should run against. This would automatically "expand" into a job for each combination of spaces described by its steps. They can say all for all spaces (dynamically discovered), a list of static space names (e.g. [2.0, 1.9]), or a list of space name filters (e.g. [feature/*] for feature branches).

Run against all branches of a repo, but only progressing one to the rest

In the simplest case, this makes it easy for me to have CI run against all branches in my side-project. It would just be a matter of adding spaces: {booklit: all} to the job:

jobs:
- name: unit
  public: true
  spaces: {booklit: all}
  plan:
  - get: booklit
    version: every
    trigger: true
  - task: unit
    file: booklit/ci/test.yml

- name: rc
  public: true
  serial_groups: [version]
  plan:
  - get: booklit
    passed: [unit]
    trigger: true
  - get: version
    params: {pre: rc}
  - put: version
    params: {file: version/version}

- name: ship
  public: true
  serial_groups: [version]
  plan:
  - get: booklit
    passed: [rc]
  - get: final-version
    resource: version
    passed: [rc]
    params: {bump: final}
  - get: release-notes
  - put: version
    params: {file: final-version/version}
  - put: booklit-rel

Run multiple versions against all branches

In more complicated cases, you may want to run against all branches, and for each branch run against different versions of the language the project is written in:

jobs:
- name: unit
  spaces:
    my-repo: all
    ruby: [2.0, 1.9]
  plan:
  - get: my-repo
  - get: ruby
  - task: unit
    image: ruby
    file: my-repi/ci/unit.yml

Propagating through a pipeline

Each combination of spaces progresses through the pipeline on its own accord. This means that if the case of ruby-1.9 above fails, ruby-2.0 can continue on its own. This seems in line with how most people would want these "dynamic pipelines" to work as it lets you get the most feedback possible for each combination. It's also how things would work if they were explicitly configured as different jobs/pipelines.

jobs:
- name: unit
  spaces:
    my-repo: [master, develop]
    ruby: [2.0, 1.9]
  plan:
  - get: my-repo
  - get: ruby
  - task: unit
    image: ruby
    file: my-repi/ci/unit.yml

- name: integration
  spaces:
    my-repo: [master, develop]
    ruby: [2.0, 1.9]
  plan:
  - get: my-repo
    passed: [unit]
  - get: ruby
    passed: [unit]
  - task: integration
    image: ruby
    file: my-repo/ci/integration.yml

Here's how this works: each resource mentioned in a plan is pinned to a single space. This space applies to all get and put steps, but also the spaces used in jobs that passed constraints refer to, when those jobs use the same resource.

This means that this:

- name: integration
  spaces:
    my-repo: [master, develop]
  plan:
  - get: my-repo
    passed: [unit]

...expands to two jobs, looking something like:

- get: my-repo{master}
  passed: [unit{my-repo:master,ruby:2.0}, unit{my-repo:master,ruby:1.9}]

- get: my-repo{develop}
  passed: [unit{my-repo:develop,ruby:2.0}, unit{my-repo:develop,ruby:1.9}]

...but then by adding a get: ruby with its own spaces, like so:

- name: integration
  spaces:
    my-repo: [master, develop]
    ruby: [2.0, 1.9]
  plan:
  - get: my-repo
    passed: [unit]
  - get: ruby
    passed: [unit]

...that results in more combinations, each with their passed constraints reduced to the same spaces that their peer get steps are pinned to:

- get: my-repo{master}
  passed: [unit{ruby:2.0,my-repo:master}]
- get: ruby{2.0}
  passed: [unit{ruby:2.0,my-repo:master}]

- get: my-repo{master}
  passed: [unit{ruby:1.9,my-repo:master}]
- get: ruby{1.9}
  passed: [unit{ruby:1.9,my-repo:master}]

- get: my-repo{develop}
  passed: [unit{ruby:2.0,my-repo:develop}]
- get: ruby{2.0}
  passed: [unit{ruby:2.0,my-repo:develop}]

- get: my-repo{develop}
  passed: [unit{ruby:1.9,my-repo:develop}]
- get: ruby{1.9}
  passed: [unit{ruby:1.9,my-repo:develop}]

Explicitly fanning in

The above behavior can be configured, and often should be in the case of shipping something, as it can otherwise quickly lead to fanning-out, something you may not want for publishing your final artifacts.

When it comes to shipping, there are a few differences:

  • You probably only want to ship the master branch, not all of them.
  • You may want to only ship things that passed all versions of ruby.

This can be done like so:

- name: ship
  spaces:
    my-repo: [master]
  plan:
  - get: my-repo
    passed: [integration: all]
  - put: rubygems

There's actually a few things to explain here. The integration: all means "get only versions from the master branch, having passed all cases of integration in which it was present". However, even if we hadn't specified it, that's also the only reasonable interpretation. We're being explicit, because there's also a case where you may want to get a single version of ruby but still do a fan-in, not just "continue" that version's sub-pipeline.

One use case for this is when you want the pipeline to have succeeded against both versions but only need one to perform some menial shipping task. In that case you'd end up with something like:

- name: ship
  spaces:
    my-repo: [master]
    ruby: [2.0]
  plan:
  - get: my-repo
    passed: [integration: all]
  - get: ruby
  - task: build-erb-release-notes
    image: ruby
    task: my-repo/ci/build-release-notes.yml
  - put: rubygems

Normally, the presence of ruby-2.0 would have selected for that space within the passed constraints of the my-repo step. By saying all, it prevents that reduction, preserving the fan-in.

How will this look in the UI?

idk lol

@Lindsayauchin halp

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Oct 12, 2017

We've pushed a spike here, demonstrating how scheduling can work with this new model.

Here's the general approach:

  1. We added a job_permutations table, which points to a jobs (id) and has the resource name -> space name mapping for its permutation of the job's spaces, as a jsonb type. There is a UNIQUE index on the job ID and mapping.
  2. builds now reference a job_permutations (id), not a jobs (id).
  3. We added a resource_spaces table, which points to a resources (id) and has the name of the space.
  4. versioned_resources now reference a resource_spaces (id), not a resources (id).
  5. At scheduling time, for each job, calculate all permutations of the job's spaces.
  6. Save the set of permutations a in the job_permutations table, via INSERT ... ON CONFLICT (job_id, resource_spaces) DO UPDATE SET active = true RETURNING id and mark all other permutations for the job_id as active = false. This way we can keep track of which spaces are active and should be scheduled, marking stale ones as inactive.
  7. The scheduler still loads up a *algorithm.VersionsDB, which now contains numbers that are resource_spaces (id) and job_permutations (id), instead of resources (id) and jobs (id).
  8. The scheduler then takes the job's input configs and turns them into algorithm input configs by tying each input to its spaced resource and generating the passed constraints by referencing the relevant permutations of the referenced job.
  9. The scheduler now computes inputs and queues pending builds for individual job permutations, so next_build_inputs and independent_build_inputs now reference a job_permutations (id), not a jobs (id).
  10. Then, the scheduler starts whatever pending builds exists with their next_build_inputs, which is now per each job permutation. This works as before.

Some of the work involved was just moving code around, since a large chunk of job responsibilities were now on job_permutations. Thankfully, we didn't really have to touch the algorithm! We renamed variables and fields and such, but the actual work it's doing is exactly the same. We could probably give it more generic names or something instead to prevent further change in the future. Or just keep renaming things to keep a consistent narrative. It's all a trade-off.

We then tried this out by hand-configuring spaces and resource versions in our database (space discovering and checking per-space isn't done yet but isn't all that interesting/risky), configuring a job with spaces: configured, and saw that the scheduler then generated the correct job permutations, created a pending build, and started each one, with the correct versions. Only gotcha is that the resource itself doesn't know about the spaces, so builds failed as e.g. their versions were referencing the wrong branch. But, the bit that we cared about worked!

One question that came up while working on this is: what does manually triggering do? Trigger every permutation? Would we move the triggering to individual permutations? This all depends on how we want to expose these things to the user, I suppose.

@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

jchesterpivotal commented Oct 23, 2017

Re: the idea

I think, right now, is I would have difficulty mapping the YAML into a mental model, or an ideal model back down to the YAML. It's possible that the GUI will create concrete representation I can hang my internal analogies off.

As a quick question, how is this different from defining more fine-grained resources? For example, release branches can be represented as separate git resources.

Re: the spike

as a jsonb type

I am uncomfortable with mixed graph/relational models, even small ones.

@JohannesRudolph

This comment has been minimized.

Copy link
Contributor

JohannesRudolph commented Oct 24, 2017

Wow! I think I haven't fully grasped every detail yet, however I can assure you that we've been there and faced exactly the challenges that this proposals aims to solve. However, I'm worried it will add a tremendous amount of complexity to the mental model of resource flows. Which no doubt could be very useful ;-)

So far, our approach for dealing with "spatial" resources, for example git-branch-heads for release, develop and feature branches, was to set up the pipeline so that we effectively have n copies of jobs, one for each spatial dimension. So effectively we have a pipeline graph with n connected graph components (of jobs and resources). We are leaning on YAML anchors to eliminate repetition.

The upside of this approach: we discovered that we actually do want different jobs for each spatial dimension (e.g. develop builds do additional stat steps etc., prod builds go to long term archival resources etc.). The existing capabilities of concourse served us well.

What we are missing though is a way to say have concourse spin up a CF space for each feature branch and tear it down once the branch is merged. We were playing with the idea of implementing a custom resource to do this, but found that it gets tricky to integrate with concourse's pipeline model. Our pipeline has just a single "graph component" of jobs shared for all feature branches. Develop/Release branch do not have that problem since they are static and we can "pre-code" them in the pipeline. However, feature branches are dynamic. The proposed model here could very well help better capture this and make it easier to pull of something like this.

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Oct 24, 2017

@jchesterpivotal @JohannesRudolph We hear your concerns about complexity and had the same concerns going into this. Personally I feel that the spike resulted in a design that actually nicely compliments today's mental model, and even reuses most of our metaphors. Scheduling, for example, remains exactly the same, except that it now schedules "job permutations", not just "jobs". The logic for progression, version selection, etc. all remains the same, only they're operating on "resource spaces", not just "resources". Yes the words are different internally, but that's all that's changed.

UI/UX was our biggest concern coming out of the spike, as we had only shown that these things can be easily expressed in .yml and behave intuitively - that's only the half of it, the user also has to understand what they've done.

I think we've made good progress on that front, thanks to @Lindsayauchin and lots of whiteboarding with @mhuangpivotal and @pivotal-jwinters.

Here's our current mockup for a "spaces"-aware pipeline UI, for jobs that have the space config {concourse-repo: [master, develop], ruby: [1.8, 1.9, 2.0]}:

image

The metaphor is fairly simple: each permutation of a job's space configuration is represented as a card. Jobs then show their cards stacked as a deck, by default. The deck can be split by resource by clicking the "*" on the resource feeding into it. This is useful for cases where you want to see clearly the pipeline run for distinct spaces (say, different IaaSes or different branches/PRs). Check out the interactive Invision prototype to see what I mean.

We would probably then want a feature to "save" the configuration for which resources are split. Not sure if that'd be an ephemeral thing or a pipeline configuration or something.

@jchesterpivotal re: jsonb, where's your proposal? ;) Seems like an odd nit to pick. It pretty straightforwardly accomplishes exactly what we want it to. This is a case where it's literally a set of permutations that we're managing, so being able to key on the set itself is very useful for upserts.

@ggeorgiev

This comment has been minimized.

Copy link

ggeorgiev commented Nov 3, 2017

This sounds like a great idea. I am looking for something like this for some time.

I am not sure that I love the hardcoded matrix in the pipeline concept though. This definitely will work for some use cases, like supported versions, but for others, it would not improve much.

The biggest challenge is with resources that have different life cycle than the pipeline. If I reiterate already mentioned example - a feature branch. For that matter, we can add release and task branches (for pr validation runs). For such I will need to redeploy the pipeline every time a new branch is added/removed ... as I have to do it right now.

Why not instead to use a similar to the resource versions model, the system to allow for obtaining the spaces dynamically?

@ggeorgiev

This comment has been minimized.

Copy link

ggeorgiev commented Nov 3, 2017

Something else to consider as well.

I have the feeling that the granularity for such feature should not be on job by job bases. For a big pipeline, I feel this will complicate the mental model of what is going on tremendously.

When I picture this feature I am imagining that the matrix items are a sub-pipelines in which you getting in and out - that shows a subset of jobs - similar to the group feature.

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Nov 3, 2017

@ggeorgiev

They can say all for all spaces (dynamically discovered), a list of static space names (e.g. [2.0, 1.9]), or a list of space name filters (e.g. [feature/*] for feature branches).

Aside from that, I don't really agree that this is a pipeline-level abstractions. Pipelines aren't a thing. They are a result of configuring jobs, which describe everything they need. The more we add to pipelines, the bigger risk in surprising behavior (i.e. a matrix applying to a job that shouldn't). Even within a single connected pipeline, at least one job will probably have a different configuration for its spaces, i.e. in order to fan-in.

@ggeorgiev

This comment has been minimized.

Copy link

ggeorgiev commented Nov 3, 2017

@vito

Sorry, I missed that part. But if the resource discovers the spaces by itself, what purpose hardcoding it in the jobs will have - an exact match filter?

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Nov 3, 2017

@timrchavez

This comment has been minimized.

Copy link
Contributor

timrchavez commented Nov 16, 2017

While this is still under development, can we get a UI mockup for how this will look with PRs (maybe even a blog that describes the two most popular use cases: branching resources (like PRs and feature branches) and matrix will work when this feature lands)? I keep telling people that a better (more intuitive) way to do PR workflows is coming to Concourse but when I describe the concept, eyes tend to glaze over. I would like to be able to show people with pictures, how the UX is changing / improving.

@jama22

This comment has been minimized.

Copy link
Member

jama22 commented Nov 16, 2017

@timrchavez unfortunately all we got is the rough UI mocks that you see above. We wanted to avoid going too deep into the UI mockup while the team was exploring the feasibility of spatial resources. A lot of the design work and discussions were focused on the yml syntax of spikes as well.

That said, when we pick this back up i think the team should focus on PRs and branching resources as a good model to design around. Lucky for us we have @Lindsayauchin and @jtarchie on the team.

Last point is, we have quite a few design prototypes in-flight that are not specifically spaces related. They are focused on re-thinking and re-building the visual language of Concourse to solve some of the issues reported in the past and to facilitate the new features like spaces and causality

@vito vito changed the title Spike: spatial resource flows Spatial resource flows Nov 18, 2017

@vito vito added the epic: spaces label Nov 19, 2017

@vito vito removed the core/build-plan label Nov 21, 2017

@codesuki

This comment has been minimized.

Copy link

codesuki commented Nov 22, 2017

The mock-up above looks fancy, but it would be nice if we could have some clickable list showing the spaces with some metadata [PR number, branch name], to be able to quickly navigate between PRs etc. Anyway looking forward to spaces, urgently needed feature imho.

@vito vito moved this from Define to Icebox in Core Nov 24, 2017

@vito vito moved this from Define to Icebox in Core Nov 24, 2017

@vito vito changed the title Spatial resource flows Epic: spatial resource flows Nov 28, 2017

@martinxsliu

This comment has been minimized.

Copy link

martinxsliu commented Sep 21, 2018

@jama-pivotal Hi James, would it be possible to get an update on the roadmap for this feature? It's something that's quite important for our usage of Concourse and it would be great to get a rough ETA on when it might be released.

I've tried reading through your weekly updates and the github issues & epic label, but I couldn't find any guidance on progress or timelines. Any insight would be greatly appreciated.

@nigeltiany

This comment has been minimized.

Copy link

nigeltiany commented Sep 28, 2018

This feature would be nice. Here are some simpler ideas for the UI.
concourse-spaces
This might help focus on a single space at a time. A button that shows all spaces in a fancy carousel is also another option if showing all spaces at once is needed.

@jama22

This comment has been minimized.

Copy link
Member

jama22 commented Oct 12, 2018

OH HAI @martinxsliu. Sorry I'm terrible at github notifications.

Also, my last comment on this issue was Jan 30 so I can see why everyone's a bit confused. Quick recap: we conducted an open call for user tests last spring/summer. In that time we experimented with different pipeline yml syntaxes and UI designs. Having worked through some of the prototypes, we realized that we needed to put that work on pause while we implement the new v2 resource interface first: https://github.com/vito/rfcs/blob/resources-v2/01-resources-v2/proposal.md This also required us to do some optimizations on resources, which we completed with #2596 #2443

And now we're working on #2651

@JohannesRudolph

This comment has been minimized.

Copy link
Contributor

JohannesRudolph commented Oct 15, 2018

Great to see that this is being worked on. I can imagine this is going to be useful. However, I just wanted to give a quick update on our use case that I described above, because it turns out.... we might not need spaces anymore :-/ ?

A quick note about our context: we're using concourse to build, test and deploy a mono-repo of ~12 microservices to 3 different production environments. We've been working for a long time with a git-flow based model using feature branches and explicit merging, but we found it doesn't scale with our team. We have very significant testing requirements and we leverage concourse to execute tests in parallel across a cluster of 6 workers to make it bearable and productive. A big problem we've had was that the amount of builds and consequently the resource consumption was going through the roof. Every change checked into the mono-repo triggered a build & test for all microservices, every merge caused a full build as well.

So we've just recently switched to a more trunk based development flow. Developers are still doing their work in feature branches that get built & tested by the CI pipeline but we no longer merge but rebase & fast-forward instead (and ocassionally developers squash their branches as well). We have built automation around automatically merging feature branches when quality gates are passed (e.g. PR review is positive, CI testing passed etc. - going to be OSS'ed soon). We also use conditional gates to determine which microservices are affected by a particular change and need to be built. This massively reduces the amount of builds & test we need to execute for a particular change in our pipeline. We have specifically developed the gate-resource to handle all the gates in our pipelines.

To make a long-story short: all branches now run through the same jobs in our CI pipeline. We no longer have the pipeline jobs effectively 3x (feature, develop, master branches) but only one instance of each.
We have an "init" job that determines what needs to be done for a particular change and then triggers the necessary parts of the pipeline. Gates track the status of a change as it progresses through our pipeline and are used to trigger other parts of the pipeline with the correct set of inputs.

This massively reduces our need for something like spaces, though I'm convinced our solution has its fair share of complexity and spaces are likely the better solution for simple "git flow + spaces" projects. I plan to make a write-up about our pipeline setup someday, but nonetheless I hope the concourse team can use this short description + what we do with gate-resource as an interesting data-point.

@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

jchesterpivotal commented Oct 16, 2018

I had a discussion with @bobcatfish about this recently and came to the same realisation as @JohannesRudolph -- Concourse's origin story as a tool for helping Cloud Foundry development explains why it's taken a while to really grip the problem of PR-driven workflows. Most of us just didn't feel the pain and given the large infusion of DNA from Pivotal Labs, had a bias to trunk-based development to start with.

@ghostsquad

This comment has been minimized.

Copy link

ghostsquad commented Oct 16, 2018

@JohannesRudolph why not separate your code base into separate repos so they can be independent (one of the main selling points of microservices)? It sounds like you have a distributed monolith, not microservices.

We've been able to mitigate the need for spaces by using Jsonnet. We still end up with pipelines with lots of duplication. But it doesn't feel like duplication when developing the pipeline, because we can produce the necessary configuration easily with jsonnet functions.

@JohannesRudolph

This comment has been minimized.

Copy link
Contributor

JohannesRudolph commented Oct 16, 2018

@ghostsquad Our decision to use a mono-repo is not primarily motivated by the architecture of the code (or the organizational structure of our teams for that matter). But let's just say we're building a bunch of "services", there's plenty of room to talk mono-repos and microservice/monoliths elsewhere ;-)

Jsonnet is really nice, we hacked & taped together ejs-compiler a while ago to do help us generate our "spatial" pipelines (just OSS'd today). What I didn't get about jsonnet back in the day when I first looked at it whether you can get it to output yaml or do you write your pipelines in the json subset of yaml?

@jchesterpivotal That sounds very interesting. I'd be curious to learn more about where you see the distinction between PR-based and trunk-based workflows in the context of building a concourse pipeline to support them? For example, I would consider our workflow still "PR-driven" in the sense that we do use Pull Requests to do code review and we build & test each PR through the pipeline. However, due to fast-forwarding changes we aim for a linear change history at the end of the day and we found it important for us to avoid expensive rebuilds when fast-forwarding, hence our experiments with gate-resource.

I see that e.g. Concourse is moving to a mono-repo too, so I'm curious what kind of workflow @vito aims for in terms of building & validating PRs? Should they live in a separate pipeline? Do you need to avoid rebuilds when merging?

@ghostsquad

This comment has been minimized.

Copy link

ghostsquad commented Oct 16, 2018

@JohannesRudolph since YAML is a superset of JSON, any valid json should work as YAML. But besides that, check out this page https://jsonnet.org/learning/getting_started.html and scrolled down to the YAML part. The only requirement for outputting to YAML, is that you output an array of documents (as seen in the example).

@thewoolleyman

This comment has been minimized.

Copy link

thewoolleyman commented Oct 16, 2018

@JohannesRudolph re: "all branches now run through the same jobs in our CI pipeline."

The problem we had with this was that it was near-impossible to see the all the builds for a given branch in isolation, because they were all mixed together in the Concourse UI history (which had issues with easily searching/paginating through history)

So, it was unusable from the perspective of wanting to see what the pass/fail history for given feature branch/PR was in Concourse - which is often very important in determining flaky builds, potential regressions, etc.

How are you dealing with or avoiding these types of problems with your approach?

Thanks,
-- Chad

@thewoolleyman

This comment has been minimized.

Copy link

thewoolleyman commented Oct 16, 2018

@ghostsquad Re:

"why not separate your code base into separate repos so they can be independent (one of the main selling points of microservices)? It sounds like you have a distributed monolith, not microservices."

I wouldn't recommend this as a solution to this problem. There's many benefits to monorepos (which is why big companies like MS, Google, and Facebook use them, and many other teams are moving to them - we did and are much happier).

These benefits of monorepos can be achieved regardless of whether or not your code ownership is split across separate dev teams or whether or not you are doing microservices or well-structured monoliths. (Pro tip: you are often better off with a well structured monolith than microservices)

Don't want to turn this thread into an unrelated bikeshed, but there's a lot more info on Google regarding this decision. Here's some:

...good summary of monorepo benefits
https://trunkbaseddevelopment.com/monorepos/

...an article and HN discussion on why you might want to use a monorepo:
https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repositories/
https://news.ycombinator.com/item?id=10007654

...an article, HN discussion, and video about Google's use of a monorepo:
https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-place/
https://news.ycombinator.com/item?id=10227000
https://www.youtube.com/watch?v=W71BTkUbdqE

HTH,
-- Chad

@JohannesRudolph

This comment has been minimized.

Copy link
Contributor

JohannesRudolph commented Oct 17, 2018

The problem we had with this was that it was near-impossible to see the all the builds for a given branch in isolation, because they were all mixed together in the Concourse UI history (which had issues with easily searching/paginating through history)

We can relate to that very well :-) I can't claim it's perfect but we get pretty good value from having each jobs report build status to bitbucket for the respective SHA being built. So if I wanted to see all the builds that ran (or failed) for a particular commit I can use bitbucket as the entry point. All build statuses posted contain a link to the respective concourse build details page, so navigation is fairly quick and usable.

What we still miss though is sort of a "progress" view that shows us how far a particular commit has made it already through the multiple stages (and even pipelines for that matter as deploy pipelines live separately to reduce the number of people with access to production credentials). This was better back in the day when we had a master, develop and feature-branches pipeline each. One could argue though that the feature branch pipeline (which did build all feature branches using git-branch-heads) has had the same problem in terms of the same jobs running for different branches.

I guess others feel the same pain with remotely complex, inter-dependent pipelines. Not sure if there's a clear answer but I've been pondering the idea of setting up a visualiser/UI on the gate repository produced by gate-resource to see the "flow" and dependencies. Right now we're contempt with the build-status.

@DanielJonesEB

This comment has been minimized.

Copy link

DanielJonesEB commented Nov 27, 2018

Potentially off-topic, but I thought I'd jot it down here whilst I'm thinking about it.

For the spatial UX, did y'all try an isometric view, and switching job rectangles to being cuboids? That'd make it easier to track interactions between spaces, as lines would be visible along the Z (spatial) axis. Lines could be rendered with 'bars' across them making it easier to grok whether the line is running along the X/Y axes, X/Z, or X/Y/Z. An orthographic projection could be used for pipelines without spaces, preserving the information density of the current UX.

@duckinator

This comment has been minimized.

Copy link

duckinator commented Feb 3, 2019

Hey, could we get an update on what the state of this is? Thanks. ^^

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Feb 3, 2019

@duckinator Sure!

So, for this to be done we need concourse/rfcs#1 finished, since that determines the way 'spaces' will be implemented by each resource type. There's a whole lot of other stuff in that RFC too, though -- it's our first big revision to the resource interface, so that increases the scope quite a bit.

The Resourcecs v2 RFC is nearing completion, and we've been making the back-end more efficient in the meantime to make the transition easier (#2386, #2596). This will be released later this month in v5.0.

With this preliminary work out of the way we can shift our focus back to Resources v2 and Spaces. Right now I can't give a super clear timeline, but we have a pretty good understanding of the problem space at this point and a good strategy for implementation, which we've learned from a bunch of proof-of-concepts and spikes.

I plan to draft an official RFC for how Spaces are used in a pipeline (similar to the mock-up shown in this issue). We didn't have an RFC process when this feature was initially thought up, and it's a huge change, so I'd like to re-introduce it through the proper channels to allow more input and collaboration from the community. Once this RFC is opened I'll probably close this issue and centralize discussion there instead. That'll make it easier to keep track of progress and it'll be a good central point to find examples and future documentation/etc.

So, things are coming along steadily - we're being careful and trying not to rush this, since resources are such a core aspect of Concourse and we have to be sure to get it right.

You can keep tabs on the work that's left in the repo projects:

https://github.com/concourse/concourse/projects

@duckinator

This comment has been minimized.

Copy link

duckinator commented Feb 3, 2019

@vito awesome, thanks for the quick response!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.