Skip to content

@concourse-bot concourse-bot released this Oct 8, 2021

✈️ Features

  • 7.4.x: add some DB optimizations (#7641) @taylorsilva 🔗
    • DB optimizations
      • Increment an event ID in-memory instead of using a Postgres Sequence
      • Only update resource config if it hasn't been updated in a minute
      • Close zstd reader in load var step

🤷 Miscellaneous

📦 Bundled resource types

Contributors
taylorsilva
Assets 14

@concourse-bot concourse-bot released this Sep 17, 2021

IMPORTANT: This release includes a few large refactors, therefore we recommend anyone using Concourse for mission-critical workflows to wait for the next few releases just in case any edge cases are found.

This release contains the version of github-release-resource that fixes the illegal base64 data at input errors from GitHub API's recent breaking change concourse/github-release-resource#108.

🚨 Breaking

  • Prefer overlay over btrfs in baggageclaim when using driver: detect (#7427) @aoldershaw 🔗

    • Previously, when the baggageclaim driver was not specified, Concourse attempts to detect the supported drivers
    • The prior driver precedence is: btrfs -> overlay -> naive
    • The new driver precedence is: overlay -> btrfs -> naive
  • Allow team members to archive pipelines (#7449) @wanderanimrod 🔗

    • Users with the member role on a team can now archive pipelines by default. The "archive pipeline" action was previously assigned to the owner role. If you've configured your own RBAC this change will not effect you.

✈️ Features

  • Removing VersionedResourceTypes from get, check and put plans (#7176) @clarafu 🔗

    • This PR is mainly a refactor but there is a behavioural change that comes along with it. If a resource uses a custom resource type, its Put, Check, Get and Task steps will now always create a check for its parent custom resource type. This check will still respect the resource checking interval and the check_every of the resource type. Because of this new feature, we no longer need to explicitly check the custom resource types in lidar.
  • Add audit information for job & pipeline pauses (#7273) @rjinskidepop 🔗

    • Add pipeline and job pause meta information - who and when.
  • Propagate groups between subpages of a pipeline (#7307) @clarafu 🔗

    • If a user was initially viewing a group in the pipeline page, this will be persisted in the pipeline breadcrumb when navigating between pipeline subpages.
  • Optimize pipeline svg rendering (#7438) @aoldershaw 🔗

    • The initial render of the pipeline page should be much faster, particularly on Chrome 92+
  • Don't query for the entire resource in the check delegate (#7474) @aoldershaw 🔗

  • Simplify atc/worker package and extract runtime abstractions (#6597) @aoldershaw 🔗

    • We will now error when a suitable worker does not exist rather than waiting forever.

🐞 Bug Fixes

  • Fix opening a link to a resource causality page (#7369) @aoldershaw 🔗

  • Don't allow empty identifiers when renaming pipelines/teams (#7370) @taylorsilva 🔗

    • Fixed a bug where a pipeline or team could be renamed to an empty string. The team/pipeline could not be deleted through fly. An error is now returned by the API if the identifier is blank
  • Sanitize prometheus metric labels (#7423) @lrstanley 🔗

    • Ensure Prometheus metric labels are valid. This resolves an issue with our bosh release, where web nodes would fail to start, due to a metric label that wasn't valid according to Prometheus.
  • Fix overlapping between inputs and jobs in UI (#7454) @xtremerui 🔗

    • Fixes an edge case that might overlap an input and job node in the pipeline view.
  • Validate if a Pipeline contains a cycle (#7455) @EstebanFS 🔗

    • The API will reject any pipelines that contains a cycle
  • Prevent open redirect to other hosts (#7459) @taylorsilva 🔗

    • Prevent an open redirect vulnerability on the /sky/login path
  • Delete btrfs volume if it exists when using the overlay driver (#7461) @taylorsilva 🔗

    • Made worker initialization more stable if you're switching from btrfs to overlay. The worker will remove the btrfs mount if it exists before creating overlay mounts
  • Fix missing label in metric concourse_steps_waiting (#7479) @Esysc 🔗

    • Fix missing label in metric concourse_steps_waiting
  • Close zstd reader in load var step (#7548) @clarafu 🔗

🤷 Miscellaneous

📦 Bundled resource types

Contributors
xtremerui, lrstanley, and 7 other contributors
Assets 14

@concourse-bot concourse-bot released this Sep 14, 2021

🐞 Bug Fixes

🤷 Miscellaneous

📦 Bundled resource types

Contributors
taylorsilva, kirillbilchenko, and aoldershaw
Assets 14

@concourse-bot concourse-bot released this Jul 29, 2021

✈️ Features

  • Fly clear-resource-cache command (#7003) @EstebanFS 🔗

    • Added fly command clear-resource-cache, you could use this following the next format
      fly -t ci clear-resource-cache -r pipeline/resource [--version some:version]
  • Build page shows name of who triggered the build in header line of build page (#7112) @evanchaoli 🔗

    • The build page now shows the username of who triggers the build if the build is triggered manually.
  • Add page to view all builds/resource versions downstream/upstream from a root resource version (#7125) @chenbh 🔗

    • Disabled by default since computing causality for large datasets can be expensive, use --enable-resource-causality or $CONCOURSE_ENABLE_RESOURCE_CAUSALITY=true to enable the web UI and API endpoint.
      • Most datasets (like the merge commit for this PR) have < 100 builds and/or resource versions and take < 100ms, but it's possible for some "slow paced" resource versions (i.e. very infrequent new versions) to generate extremely large datasets
      • There is an automatic cutoff at 5000 builds or 25000 resource versions. On our deployment, the call for our slowest paced resource took about ~7 seconds to process, most of which is spent in the DB query
    • The causality page can be navigated to from the resource page
      Screen Shot 2021-06-03 at 11 37 08 AM
    • The causality page displays all the builds and resource versions that was generated from (downstream) or resulted in (upstream) the creation of a particular resource version
      Screen Shot 2021-06-03 at 11 25 03 AM
    • The downstream graph will put the root resource version on the left whereas the upstream graph will put it on the right
    • It takes into account all the intermediate resource versions when computing the final graph. In the picture above, while the resource page only shows that git version: 123 is a direct input to integrate #4 & #5, there is also an indirect link from git version: 123 -> test #19 -> ... -> intermediate-3 version:123 -> integrate #6 & #6.1
  • Support soft policy enforcement (#7139) @evanchaoli 🔗

    • This feature doesn't break the existing OPA policy check. If you have enabled OPA policy check, and you don't need "soft" policy enforcement, then you just don't need to do any configuration change.
    • 3 new ATC cli options are added:
      • CONCOURSE_OPA_RESULT_ALLOWED_KEY: specifies a key of allow flag in OPA returned result
      • CONCOURSE_OPA_RESULT_SHOULD_BLOCK_KEY: specifies a key of should-block flag in OPA returned result
      • CONCOURSE_OPA_RESULT_MESSAGES_KEY: specifies a key of messages in OPA returned result

    For example, if OPA returns the following result:

    {
        "result": {
            "allow": true,
            "block": true,
            "reasons": ["foo", "bar"]
        }
    }

    then CONCOURSE_OPA_RESULT_ALLOWED_KEY should be set to result.allow; CONCOURSE_OPA_RESULT_SHOULD_BLOCK_KEY should be result.block, and CONCOURSE_OPA_RESULT_MESSAGES_KEY should be result.reasons.

    NOTE: allow and block in OPA result should be boolean type, because it's easy to convert other types to boolean in an OPA policy.

  • Add ability to comment on a build (#7147) @multimac 🔗

    • You can now leave comments on builds. For instance, this can be used to give context to your coworkers about why a particular build failed:
      Screen Shot 2021-06-30 at 5 40 45 PM

    • If a build has a comment, it is displayed with a small marker to help you quickly find builds of interest. Hovering over the build displays a portion of the comment:
      Screen Shot 2021-06-30 at 5 41 52 PM

  • Add teamName to concourse_steps_wait_duration metrics (#7154) @Esysc 🔗

  • Use browser cache API for dashboard caching (#7247) @aoldershaw 🔗

    • The cached API responses on the dashboard no longer need to get truncated, which was previously introduced to work around localStorage limits
  • Allow interpolation in the across step values (#7252) @aoldershaw 🔗

    • The across step now supports dynamic interpolation of values. For instance, this can be combined with the set_pipeline step and instanced pipelines to set a dynamic list of pipelines:
      - load_var: branches
        file: branches/branches.json
      - across:
        - var: branch
          values: ((.:branches))
        set_pipeline: my-app
        file: ci/pipelines/my-app.yml
        instance_vars: {branch: ((.:branch))}
  • Cache the list of workers in memory (#7268) @aoldershaw 🔗

    • Scheduling containers should be more performant by reducing the number of required database calls
  • Optimize build log collector (#7327) @evanchaoli 🔗

    • Optimized a SQL statement used to remove build logs. This optimization will specially benefit large deployments that have a lot of pipelines.
  • Enable emitting dogstatsd metrics over uds (#7338) @jmhwang7 🔗

    • The Datadog emitter can now be configured to communicate with the Datadog agent over Unix Domain Sockets

🐞 Bug Fixes

  • containerd: properly populate /etc/hosts and /etc/hostname (#7041) @muntac 🔗

    • containerd: /etc/hosts and /etc/hostname are correctly populated
  • Handle 403 for vault preflight check of V2 (#7057) @xtremerui 🔗

  • atc: across step logs errors (#7090) @taylorsilva 🔗

    • Across step emits an error event when one of the sub-steps errors
  • containerd: Mount /dev/fuse to privileged containers (#7098) @aoldershaw 🔗

  • atc(fix): fixed a bug in resource check rate limiter. (#7102) @evanchaoli 🔗

    • Fixed a bug in check rate limiter that caused slow checks.
  • fix BaseResourceType for streamed volumes (#7108) @vito 🔗

  • Fix worker restart issue with containerd daemon and beacon (#7113) @muntac 🔗

    • Fix worker stall issue when restarting with containerd. Exit the worker's beacon process gracefully if any other top level process like the containerd daemon fails. Wait for containerd daemon to come up before starting the containerd Garden server.
  • Fix memory leak in notification bus (#7120) @aoldershaw 🔗

  • containerd: default to root if /etc/passwd is missing (#7124) @aoldershaw 🔗

    • Fixes a regression introduced in 7.3.0 that prevented containers that don't have an /etc/passwd file from running
  • Fix algorithm considering reruns as new builds (#7144) @taylorsilva 🔗

    • Fixes pipelines getting stuck with the same inputs when a job upstream of a job with version: every succeeds and is rerun
  • containerd: keep tasks running after concourse worker restarts gracefully (#7148) @aoldershaw 🔗

    • The containerd runtime is now more resilient to the concourse worker process gracefully restarting (e.g. via monit restart)
      • Tasks that were started prior to restart will continue to run when the worker process comes back up
      • This matches the behaviour of the Guardian runtime
  • Fixed build log reaper not respecting when both Days and Builds are set (#7179) @EstebanFS 🔗

    • The build log reaper has two options for determining when to reap logs. Before, if both of the options are set, it would reap if either of the two options were true, rather than requiring both of them to be satisfied
  • Apply a minimum rate limit for resource checking (#7218) @aoldershaw 🔗

    • If CONCOURSE_MAX_CHECKS_PER_SECOND is unset, Concourse will try to distribute checks evenly over the course of the check interval to reduce the concurrent load on external systems.
    • If there are few resources in a Concourse deployment (~1-20), checks may have to wait a substantial amount of time to run in order to space the checks out evenly. However, there's no real benefit to doing this, since having just a few resources doesn't cause significant load in the first place.
    • Now, Concourse ensures that at least one check is allowed to run per second
  • atc/db: prevent creation of duplicate check builds (#7221) @taylorsilva 🔗

    • Prevent duplicate checks from being created for a single resource
  • Fix browser back button after selecting a group (#7249) @aoldershaw 🔗

    • Previously, if a pipeline group was selected in the UI, the back button would not work (you'd have to press it twice to go back)
  • set_pipeline unpauses previously archived pipelines (#7255) @aoldershaw 🔗

    • When an archived pipeline is un-archived via the set_pipeline step, it will be unpaused
  • GC task caches belonging to archived pipelines (#7272) @aoldershaw 🔗

  • containerd: Clean up networking files in /tmp (#7276) @taylorsilva 🔗

    • Fixed a bug where the containerd runtime would create networking related files under /tmp and never delete them. They are now made under the --work-dir set for the worker and are cleaned up when the container is deleted. You can delete any lingering network files under your workers /tmp directory after upgrading.
  • Fix prometheus emitter not setting default attributes (#7294) @chenbh 🔗
    Additional metrics attributes configured by --metrics-attribute now propagates to the prometheus emitter correctly.

  • run check builds GC in batch (#7323) @xtremerui 🔗

🤷 Miscellaneous

📦 Bundled resource types

Contributors
vito, multimac, and 11 other contributors
Assets 14

@concourse-bot concourse-bot released this Jul 27, 2021

✈️ Features

  • Optimize build log collector (#7334) @evanchaoli 🔗
    • Optimized a SQL statement used to remove build logs. This optimization will specially benefit large deployments that have a lot of pipelines.

🐞 Bug Fixes

🤷 Miscellaneous

  • Bump otel to 0.20.0 (#7305) @xtremerui 🔗
    • Bump opentelemetry to v0.20.0 to address CVE in one of its dependancies (apache/thrift)

📦 Bundled resource types

Assets 14

@concourse-bot concourse-bot released this Jun 15, 2021

🐞 Bug Fixes

🤷 Miscellaneous

📦 Bundled resource types

Assets 14

@concourse-bot concourse-bot released this May 28, 2021

🐞 Bug Fixes

  • Bump guardian to 1.19.28 🔗
    • Fixes a bug where guardian would fail to start up when the kernel version contained an unexpected suffix

🤷 Miscellaneous

📦 Bundled resource types

Assets 14

@concourse-bot concourse-bot released this May 25, 2021

🚨 Breaking

  • Bump opentelemetry to 0.19.0 (#6787) @aoldershaw 🔗

    • The service name Honeycomb tracing exporter is now configured via the more general --tracing-service-name (CONCOURSE_TRACING_SERVICE_NAME) rather than --tracing-honeycomb-service-name (CONCOURSE_TRACING_HONEYCOMB_SERVICE_NAME)

✈️ Features

  • Cache streamed volumes and use local cache when looking for volumes (#6660) @evanchaoli 🔗
    Optimize resource cache streaming and get step.

    • Mark streamed resource cache volumes as resource cache, to avoid duplicate streaming in next runs.
    • If a resource from a get can be found on some workers, then get step will do nothing. This will reduce times of Concourse connecting to external systems, such as git, docker hub, and so on.
    • This feature is currently opt-in and can be enabled using CONCOURSE_ENABLE_CACHE_STREAMED_VOLUMES flag.
  • Re-ordering instanced pipelines (#6830) @EstebanFS 🔗

    • Instanced Pipelines are allowed to be re-ordered with in their group through the UI (using the drag and drop functionality) or using the fly command:
      fly -t dev oip -g groupName -p key1:var1 -p key2:var2
  • Enhance syslog-drainer to make it more useful (#6834) @SimonXming 🔗

    • Add event_id into syslog-drainer entries, to get the correct order of "drained" build logs.
    • Add more supported event_type for syslog-drainer to include more info for "drained" build logs.
  • Enhance webhook triggered checks (#6854) @evanchaoli 🔗

    • When multiple pipelines hold a common resource and webhook calls against the common resource, checks are sent to all pipelines at same time. Without this enhancement, each webhook call will cause a check to run. With this enhancement, only a single check will run, which is the expected behavior as a global resource.
  • Allow override of container limits in task config (#6867) @BooleanCat 🔗

    • Pipeline authors can now set container_limits for reusable tasks in pipelines. Any limits set in the pipeline will override the limits set within the reusable task file.
  • Use cursor-based pagination for build events (#6873) @aoldershaw 🔗

    • Optimizes fetching build logs from the DB for builds with massive logs
  • Use display_user_id field to render username in web interface (#6970) @logyball 🔗

  • Set Content-Security-Policy and Cache-Control Headers (#6949) @taylorsilva 🔗

    • A Content-Security-Policy header is now set with a default value that will block framing of the Concourse web UI. This was already possible with the default value of the X-Frames-Option header.
      • The CSP header value is configurable with CONCOURSE_CONTENT_SECURITY_POLICY
    • A Cache-Control header is set on every page with a default value of no-store, private. The value of the header is overwritten for some paths (i.e. web assets)

🐞 Bug Fixes

  • Ensure stdin never errors when using containerd with TTY enabled (#6791) @chenbh 🔗

    • Fixed bug with containerd runtime where builds to error out if it runs for a long time without any output
  • Add trigger for deleting pipeline (#6880) @xtremerui 🔗

    • Fix a bug that might leave orphan pipeline_build_events_* table in DB when deleting a team. Pipelines belong to the deleted team will be destroyed by DELETE CASCADE but associated events table was not cleaned up properly.
  • Fix volume GC query to not include volumes with children (#6902) @xtremerui 🔗

    • Fix query that causes volume cannot be destroyed as children are present in web and update or delete on table "volumes" violates foreign key constraint "volumes_parent_id_fkey" in DB.
  • Set autocomplete to off for login form (#6920) @taylorsilva 🔗

    • add autocomplete="off" to the top-level form and username tags.
  • Scan unchecked resource-types (#6923) @EstebanFS 🔗

    • Fixed an edge case where a put-only resource's parent-type would not be checked
  • Ignore "not found" error on process deletion for Containerd runtime (#6959) @aoldershaw 🔗

  • worker: Set PATH based on UID instead of container's privileged state (#6982) @taylorsilva 🔗

    • Containerd: fixed a bug where PATH did not contain directories to system tools (i.e. /sbin) when a user/process was root. Only effects unprivileged containers.
  • Fix Postgres deadlock when frequently setting pipelines (#7011) @aoldershaw 🔗

  • containerd: allow use of non-existent uids (#7029) @muntac 🔗

    • containerd supports running images with non-existent UIDs such as distroless images.

🤷 Miscellaneous

📦 Bundled resource types

Assets 14

@concourse-bot concourse-bot released this Apr 14, 2021

🚨 Breaking

  • Wait for worker matching strategy when scheduling build steps (#6635) @multimac 🔗
    • Previously, if no workers satisfied the container placement strategy for a step (with the exception of task steps when using the limit-active-tasks placement strategy), the step would simply error the build
    • Now, all steps will wait for a worker to become available
    • The metric concourse_tasks_waiting was removed and replaced with concourse_steps_waiting{type="task"}

✈️ Features

  • Add ability to navigate to resources page from build page (#6662) @chenbh 🔗
    UI: clicking on the version text for a get/put step in the Build page will now navigate directly to the Resource page with the corresponding version expanded

  • Allow using LDAP as a password connector (#6671) @aoldershaw 🔗

    • By setting --password-connector ($CONCOURSE_PASSWORD_CONNECTOR) to ldap, you can authenticate to Concourse with fly login -u ... -p ... using your LDAP credentials
      • Enabling this feature prohibits the use of local users
    • If you use an attribute other than username for authenticating with LDAP (e.g. email address), you can now configure --username-prompt ($CONCOURSE_USERNAME_PROMPT) to change the help text when logging in via the UI
  • Optimize check creation in DB (#6845) @aoldershaw 🔗

  • Add DB index to optimize paginating job builds (#6871) @aoldershaw 🔗

  • enhance put.inputs detect to ignore prefixed . and .. (#6705) @evanchaoli 🔗

    • input: detect now can handle paths prefixed by . and ...

🐞 Bug Fixes

  • Fix empty worker tags (#6057) @aholyoake-bc 🔗

  • runtime: check if swap limits is enabled (#6652) @taylorsilva 🔗

    • The containerd runtime will conditionally set memory swap limits if it detects that memory swap limits are enabled
  • runtime: timeout set to 0 means there is no timeout (#6655) @EstebanFS 🔗

    • When CONCOURSE_CONTAINERD_REQUEST_TIMEOUT is set to 0 that means there is no timeout
  • feat(atc): add check build metrics. (#6656) @evanchaoli 🔗

    • Fixed metrics BuildsStarted, BuildsRunning, BuildStarted, BuildFinsished to exclude check builds.
    • Added check build metrics: CheckBuildsStarted, CheckBuildsRunning, CheckBuildStarted, CheckBuildFinsished
  • better handling for containerd error message (#6668) @muntac 🔗

    • Fixed a bug with the containerd runtime where gracefully stopping a container might have failed with an unhandled error. Now it gracefully shuts down.
  • Prevent UI from stalling when you keep the resource page open for a while (#6703) @aoldershaw 🔗

  • move migration table updating SQL into a migration transaction (#6727) @xtremerui 🔗
    Fix a bug where a completed migration was not recorded in migrations_history table

  • Build image resource caches foreign key constraint to job ids should be on delete cascade (#6757) @clarafu 🔗

    • This change fixes a bug that was introduced in v7.1.0 where deleting a pipeline could possibly result in a 500 error. This was caused by a foreign key constraint within the build_image_resource_caches table referencing a job in the jobs table.
  • Fix race condition in containerd runtime resulting in lost output for quickly printing-then-exiting processes (#6776) @vito 🔗

  • update check metrics comments. (#6858) @evanchaoli 🔗

    • Just update code comments, no release impact.

🤷 Miscellaneous

📦 Bundled resource types

Assets 14

@concourse-bot concourse-bot released this Mar 29, 2021

🐞 Bug Fixes

  • backport #6197: Prevent retrying on worker error when build is aborted (#6598) @evanchaoli 🔗

  • Bump lib/pq to 1.10.0 which fixes a regression in lib/pq where under certain circumstances the driver would not drop dead connections and never recover. (#6746) @taylorsilva 🔗

  • Fix a panic in the New Relic metrics emitter (#6747) @taylorsilva 🔗

🤷 Miscellaneous

📦 Bundled resource types

Assets 14