Create an Easier Way to Display Build Data by Team for Main/Admin Users

[scottietremendous]

Summary

For auditing and/or billing purposes, there will be times when Operators want to see build data across all team during a given duration.

Currently, Operators (defined as users logged in under admin/main) can login to any team within their Concourse and see a summary of their build data in sequential order starting with the most recent.
ex. fly -t wings -t whatever-team -c 100

The current model is limiting for Operators running large-scale Concourse Instances for a few reasons:

• They have to login to each team individually when often times they'll want a summary of every team's build data sorted by team
• They'll want to be able to choose a specific duration of time (every build over a given week, month, etc.).
  • For instance, with our current set-up, if a user wanted to isolate build data to a specific date in order to identify a high usage team/pipeline, they'd need to set a wide net of builds via --count in work their way down to find the date they had in mind.

Proposal

I believe we should add more robust filtering to builds. The two I have in mind are:

• The ability to display a snapshot of build data across all teams for users logged in under main/admin
• The ability to set a duration of time (start/end) and display all build data within that given duration

Any other filtering ideas for builds are more than welcome!

[cirocosta]

Hey @scottietremendous ,

@YoussB and I were thinking about some possibilities for this feature, coming up with some suggestions for each reason pointed out:

The ability to display a snapshot of build data across all teams for users logged in under main/admin

NO MODIFICATIONS TO fly

    We could leverage the fact that `fly` is already capable of listing
    teams and builds for a team.

    That means that with 0 modifications to `fly`, we can retrieve all builds
    from all teams with something in the lines of:

       ./fly -t wings teams --json | jq -r '.[] | .name' | xargs -I [] ./fly -t wings builds -t []
         |                    |      |                 |      |                                  |
         *--------------------*      *----------------*       *----------------------------------*
          list all teams but in    capture only the name       for each entry, execute `fly builds`
          JSON format                                           with the team name.


INCORPORATING THE BEHAVIOR IN fly:

  We could add some flags to it in order to do what we want:

  - filtering builds by a regex:

      fly -t wings builds --filter-teams='cf-*'

  - specifying each team

      fly -t wings builds --teams='team1,team2,team3,..
        (actually, maybe w/ go-flags that would mean
        `fly -t wings builds --team=team1 --team=team2 --team=team3 ...`)

  - all teams

      fly -t wings builds --all-teams

The ability to set a duration of time (start/end) and display all build data within that given duration

We could add two flags that delimit the range in which builds are
returned.

fly -t wings builds \
  --from=<starting_date> \
  --to=<ending_date>

Using that with the first proposal (piping teams to xargs with fly builds), we would be able to essentially list all builds in a given time range from all teams.

Wdyt?

Thx

[scottietremendous]

@cirocosta - I really like this solution since it doesn't reinvent the wheel. It should be easy to pick-up right away within the cli. How do you see the output being formatted in the command line when multiple teams are selected? It would require a change to the current table format of:
id | pipeline/job | build | status | start | end | duration

[cirocosta]

Hey,

Here's our TODO Checklist:

• Check if atc supports the queries we need.
  • since and until (by time); 👎
    • all builds
    • per-pipeline builds
    • per-job builds
    • per-team builds
  • by team 👍
• Document the changes in the API here in the issue
• Check if go-concourse has the parameters for the query types:
  • since and until (by time) 👎
    • it does since and until by id, not time :(
  • by team 👍
• Add some test flight tests.
  • fly builds
• Add integration tests.
  • fly builds
• Add Unit tests.
  • atc's db package
• Check if we can use a query without teams.
• Check access permissions 👍
• implement --team <team_name>
• rename old --team as --current-team
• implement --all-teams
• implement the idea of a "range" by time
  • --since
  • --until
• Include team column when multiple teams return

[cirocosta]

Hey,

We've been wondering about making the builds command look like:

fly -t TARGET builds --current-team --team EXTRA_TEAM_1 --team EXTRA_TEAM_2 ...

This changes the current --team flag into --current-team and --team would be used to specify filtered teams.

wdyt?

cc: @jama-pivotal @scottietremendous

[YoussB]

Changes added per this issue:

• added --team flag to filter by team(s). the user using this command should be either an admin, or a part of each team they are trying to get builds for.
• the old --team was changed to --current-team which is a boolean that get only builds for the current team.
• added --all-teams flag to get builds for all available teams for the current user.
• for all the above flags as well as the old flags --job & --pipeline, the user can specify a time range for the builds to be returned in the form of --since=2006-01-02 15:04:05 --until=2007-01-02 15:04:05