Permit overlapping inputs, outputs, and task caches

[vito]

Feature Request

What challenge are you facing?

It's pretty convenient in some cases to be able to have an output directory reside underneath an input, or be the same as an input, e.g. when making a commit to a repo.

This was a limitation from Garden-Linux that probably doesn't apply anymore with Garden-runC. As long as the mounts are given in the right order.

A Modest Proposal

Let's uh...allow it? Just remove the validation and see that it works.

One gotcha: if an input is the exact same path as an output, we shouldn't mount over it. We can probably just use the input COW volume as both the input and the output and not make a volume just for the output.

[codesuki]

My use case would be the following.
I have several build steps that all build different docker images, later I want to all upload them somewhere from one task. Now I need to collect them from different output folders. Would be nice if the output paths would be shared and you could add things to it.

[cirocosta]

Hey,

During the development of this feature, we were unsure about one use case: when two inputs overlap.

For instance, consider the following plan:

concourse/testflight/fixtures/volume-mounting.yml

Lines 208 to 230 in bf98b61

	- name: input-same-input
	plan:
	- get: some-resource
	- get: other-resource
	- task: step1
	config:
	platform: linux
	image_resource:
	type: mock
	source: {mirror_self: true}
	inputs:
	- name: some-resource
	- name: other-resource
	path: some-resource
	run:
	path: sh
	args:
	- '-ec'
	- |
	cat some-resource/other_file

There we have two separate resources pointing to the same path some-resource. Intuitivelly, this would mean that we'd have two volumes pointed to the same place:

VOLUMES         CONTAINER DIR TREE


 vol1 -----.
           |
           +--> /tmp/build/<build-id>/some-resource
           |
           |
 vol2 -----*

Given that we'd not like to stack those mounts, we end up picking one of them (based on alphabetical ordering - not necessarily in the order the inputs provided).

Do you think that's an interesting case to support?

thx!

[vito]

I think picking one is fine. I wouldn't expect anything else tbh. We could add a validation for that but it's no big deal; it seems somewhat unlikely to happen in the real world.

…

On Wed, Oct 10, 2018, 2:42 PM Ciro S. Costa ***@***.***> wrote: Hey, During the development of this feature, we were unsure about one use case: when two inputs overlap. For instance, consider the following plan: https://github.com/concourse/concourse/blob/bf98b6129ff5956e24cc6f4d1bbc44159194d430/testflight/fixtures/volume-mounting.yml#L208-L230 There we have two separate resources pointing to the same path some-resource. Intuitivelly, this would mean that we'd have two volumes pointed to the same place: VOLUMES CONTAINER DIR TREE vol1 -----. | +--> /tmp/build/<build-id>/some-resource | | vol2 -----* Given that we'd not like to stack those mounts, we end up picking one of them (based on alphabetical ordering - not necessarily in the order the inputs provided). *Do you think that's an interesting case to support?* thx! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1799 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHWEl_rEndvEayxOfShX-mmo9yg2qvks5ujj-xgaJpZM4QW4p-> .

[jamie-pate]

One thing that's unclear from the docs is how this affects input_mapping/output_mapping, whether the input names are the same or just the paths, etc... (there's no examples of this use case that I can find)

This should work as expected right?

- get: git-src # some git resource
  version: every
  trigger: true
- task: prep
  file: git-src/prep_task.yaml
  input_mapping: {src: git-src}
  output_mapping: {src: prepped-src}
- task: test # imagine this uses multiple parallel tests or a separate container or something
  file: prepped-src/test_task.yaml # this fails trying to find the yaml file?
  input_mapping: { src: prepped-src }

prep_task.yaml:

---
# ... configure platform/image_resource etc
inputs: # is it ok to just have the same name or do they need the same path with different names?
- name: src
outputs
- name: src 
run:
  dir: src
  path: prep.sh

[jamie-pate]

One of my issues seems like it's causde by caches combined with this overlay... (the cached content disappears after the task ends obv)