New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for purging version history of a resource. #145

Closed
topherbullock opened this Issue Sep 16, 2015 · 16 comments

Comments

@topherbullock
Copy link
Member

topherbullock commented Sep 16, 2015

Currently there is no way to clean out the version history of a resource outside of re-creating it under a different name. This would be useful if the state of a resource changes outside of concourse and the reference history is no longer true, or for rolling back the version resource to the initial state.

@concourse-bot

This comment has been minimized.

Copy link

concourse-bot commented Sep 16, 2015

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

  • #103534396 Support for purging version history of a resource.

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

@jamesjoshuahill

This comment has been minimized.

Copy link

jamesjoshuahill commented Sep 14, 2016

+1

@topherbullock

This comment has been minimized.

Copy link
Member Author

topherbullock commented Feb 1, 2017

@vito

make it happen

@topherbullock topherbullock self-assigned this Feb 2, 2017

@devdavidkarlsson

This comment has been minimized.

Copy link

devdavidkarlsson commented Mar 2, 2017

#913 duplicate-ish

@topherbullock

This comment has been minimized.

Copy link
Member Author

topherbullock commented Mar 2, 2017

@vito

make

it

happen

@devdavidkarlsson

This comment has been minimized.

Copy link

devdavidkarlsson commented Mar 3, 2017

This is such a horrible bug for me 🤕 just disabling the version leaves the jobs stuck in pending...

@craigfurman

This comment has been minimized.

Copy link

craigfurman commented Mar 23, 2017

@vito is there any chance of this being prioritized soon? Especially after #372 was closed ;)

@jamesjoshuahill

This comment has been minimized.

Copy link

jamesjoshuahill commented Mar 23, 2017

Please please please 🐬

@joshzarrabi

This comment has been minimized.

Copy link
Contributor

joshzarrabi commented Apr 6, 2017

I also I agree that this would be a good thing.

@vito

This comment has been minimized.

Copy link
Member

vito commented Apr 7, 2017

Questions I have around this (not shutting it down, we just need to define the semantics):

  • What happens to the data builds have referencing particular versions as inputs / outputs?
    • Do we want these truly gone in that case, or somehow marked "deleted"? How does this relate to the later story around resources supporting a delete action (#534)? If at all?
  • Is it really just 'delete all' or would 'delete individual/range' be useful? (Are there implications for which route we take?)
    • We'll need to figure out the UI/UX for this based on the decision we make here.
  • Is the need for this really pointing at a gap in the story around resources? Should it be possible to get in this state? For example, changing a resource's config could clear things out, or check could be used to "sync" with the source of truth, or maybe changing a resource's config forces that sync, etc. etc.

The risk in adding this is similar to our "knobs" anti-pattern (which we'll hopefully have a doc for soon), i.e. having controls that require constant/periodic attention from users in order for Concourse to continue functioning correctly.

@devdavidkarlsson

This comment has been minimized.

Copy link

devdavidkarlsson commented Apr 7, 2017

Marked as deleted and not used in new builds would work in my case, If it then would force the pull of a new resource that is.

I thought the power button on the resource versions would have this effect but they did not.

@jamesjoshuahill

This comment has been minimized.

Copy link

jamesjoshuahill commented Apr 10, 2017

Hi @vito, thank you for these excellent questions!

I think delete and delete range (e.g. all) actions would have been useful on the occasions I have wished for purge. Here are two scenarios that might help discussion:

Testing jobs that bump semver in a version resource

Once you've bumped the version you cannot go back :o( In this case, only the latest version is needed. So the actions I imagined were:

  • pause the pipeline
  • set initial version in version resource
  • purge version resource history
  • restart the pipeline

If I deleted the semvers used during testing then I presume I would not be able to use them again. So I would have to remember to set an arbitrarily high test version before I started testing jobs to avoid collisions with deleted versions later. Maybe it would be simpler to purge, i.e. forget, those versions of the resource and the builds associated with them?

Tag moved on a git resource

Once a worker has pulled a git resource it cannot detect a tag when it is moved. Whilst moving git tags raises other problems, it does happen. In this case, I wanted to flush the cache of the git resource, which is more like the sync idea you suggested.

@vito vito removed the unscheduled label May 8, 2017

chendrix added a commit that referenced this issue May 17, 2017

bump fly
Submodule src/github.com/concourse/fly 6a64d48..510483c:
  > Merge pull request #145 from jmcarp/target-completion

clarafu added a commit that referenced this issue May 18, 2017

bump atc fly testflight
Submodule src/github.com/concourse/atc ddfeb64..7f4b7ba:
  > move NewGCM to atccmd
  > moves get build and builds in jobserver to use dbng
  > renames gcng as gc
  > add encryption to job config
  > update bindata for elm
  > update wording of legend
  > Adding chart for dotted and solid lines in pipeline
  > show metadata when resource cache is used
  > Merge branch 'etaty-patch-1'
  > Merge pull request #151 from jmcarp/path-completion
Submodule src/github.com/concourse/fly 6a64d48..510483c:
  > Merge pull request #145 from jmcarp/target-completion
Submodule src/github.com/concourse/testflight bf57b8a9e..5b5b2fc5a:
  > use unique "another-pipeline" names
  > fix testflight compilation failures
  > add tests for resource cache metadata
  > extract pipeline helpers to helpers package

Signed-off-by: Andrew Edstrom <aedstrom@pivotal.io>

@vito vito added this to Research in Core Aug 12, 2017

@vito vito moved this from Define to Review in Core Aug 14, 2017

@vito vito added core/api and removed core: lifecycle labels Oct 2, 2017

@vito vito removed the enhancement label Nov 28, 2017

@vito vito moved this from Review to Icebox in Core Nov 28, 2017

@vito vito added the enhancement label Dec 8, 2017

@spikymonkey

This comment has been minimized.

Copy link

spikymonkey commented Mar 7, 2018

We just been stung by this after cloning a pipeline to start building from a new release series branch (1.5.x), and accidentally leaving a version resource pulling from a version file in the old git branch (master). As the versions in master advanced to 2.0.0 and beyond, they were pulled into our new 1.5.x builds... and as others have found, there's no good way back from this. We've ended up renaming the version resource and input_mapping it in all of our tasks, which adds unnecessary complexity to our pipeline. It'd be great to have a way to fix this kind of issue that don't involve making permanent changes to the pipeline.

@krishicks

This comment has been minimized.

Copy link
Contributor

krishicks commented May 24, 2018

I had a lot of trouble with this while developing a resource; the versions weren't coming back in the right order which meant each time I wanted to try something different I had to create a new resource.

I also had to create a new tag each time because there's no way I can tell when Concourse has actually picked up the new version. Docker itself seems to be slow at reporting new versions after a push, so there's some bit of time I'd have to wait before I could even do the rename of the resource to get a fresh set of versions.

Also, now that I've got the resource working again, all the old names I used for the resource are effectively dead because there are badly ordered versions in their history such that they'll never see a new version again.

@arbourd

This comment has been minimized.

Copy link

arbourd commented Jun 16, 2018

I came across this today and found a somewhat temporary solution -- destroy the entire pipeline ;) Obviously, this won't work for everyone.

@vito vito moved this from Icebox to Backlog in Core Jul 16, 2018

tcdowney added a commit to cloudfoundry/capi-ci that referenced this issue Aug 23, 2018

Use cf-deployment-concourse-tasks/run-cats to run CATS
The cats-concourse-task task that we were using to run
acceptance tests in our environment was deprecated in March
and is packaged with an older version of the CF CLI

A recent change to CATS requires support for `cf tail` so
we need to use the recommended run-cats task from
cf-deployment-concourse-tasks

Additionally, the version tag filtering on the
cf-deployment-concourse-tasks resource was removed while
troubleshooting this issue so it started pulling in newer versions
of that resource which are incompatible with our current
environments. Since there is not an easy way to clear fetched
versions from a resource we had to rename it.

See: concourse/concourse#145

Signed-off-by: Eli Wrenn <ewrenn@pivotal.io>
@clarafu

This comment has been minimized.

Copy link
Contributor

clarafu commented Dec 19, 2018

This issue is resolved through the completion of #2386 because resource versions are no longer tied to a resource name, but rather through it's config. For example, each time the source of a resource is changed, it will produce new versions. If we had a pipeline with a resource:

resources:
- name: some-resource
  type: git
  source:
    repository: concourse/concourse

and we set the pipeline again after changing the source:

resources:
- name: some-resource
  type: git
  source:
    repository: concourse/other-concourse

the version history would be reset, containing only the versions for concourse/other-concourse.

Essentially, we did not add support for purging resource history, but instead have resource versions depend on it's resource config (resource type and source) instead of the resource name.

@clarafu clarafu closed this Dec 19, 2018

Core automation moved this from Backlog to Done Dec 19, 2018

@vito vito added this to the Resources v2 milestone Jan 7, 2019

@vito vito added this to Planned in Resources v2 via automation Jan 8, 2019

@vito vito moved this from Planned to Done in Resources v2 Jan 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment