flagging a input Directory as writable still symlinks all files, not copy

[gijzelaerr]

hi!

When I define an input folder as writable it is still being symlinked. Example:

cwlVersion: v1.0
class: CommandLineTool

requirements:
  - class: DockerRequirement
    dockerFile: |
       FROM kernsuite/base:1
       RUN docker-apt-install aoflagger
    dockerImageId: vermeerkat/aoflagger
  - class: InlineJavascriptRequirement
  - class: InitialWorkDirRequirement
    listing:
      - entry: $(inputs.ms)
        entryname: flagged.ms
        writable: true

baseCommand: "aoflagger"
arguments: ["flagged.ms"]

inputs:
  ms: Directory

outputs:
  ms:
    type: Directory
    outputBinding:
      glob: flagged.ms

But if I then look inside the intermediate cached results, the recursive directory structure is created but all files are symlinked:

gijs@sherlock:~/vermeerkat-cwl/cache/085d5148d1d46592125b34efa141b372/flagged.ms (master)
λ  ls -al
total 132
drwxr-xr-x 15 gijs gijs 4096 Jan 30 11:30 .
drwxrwxr-x  3 gijs gijs 4096 Jan 30 11:30 ..
drwxr-xr-x  2 gijs gijs 4096 Jan 30 11:30 ANTENNA
drwxr-xr-x  2 gijs gijs 4096 Jan 30 11:30 DATA_DESCRIPTION
drwxr-xr-x  2 gijs gijs 4096 Jan 30 11:30 FEED
drwxr-xr-x  2 gijs gijs 4096 Jan 30 11:30 STATE
lrwxrwxrwx  1 gijs gijs   84 Jan 30 11:30 table.dat -> /home/gijs/vermeerkat-cwl/cache/ca8d89e0b4b1c166a06924f8a8218b42/result.ms/table.dat
lrwxrwxrwx  1 gijs gijs   83 Jan 30 11:30 table.f0 -> /home/gijs/vermeerkat-cwl/cache/ca8d89e0b4b1c166a06924f8a8218b42/result.ms/table.f0
lrwxrwxrwx  1 gijs gijs   83 Jan 30 11:30 table.f1 -> /home/gijs/vermeerkat-cwl/cache/ca8d89e0b4b1c166a06924f8a8218b42/result.ms/table.f1
lrwxrwxrwx  1 gijs gijs   83 Jan 30 11:30 table.f2 -> /home/gijs/vermeerkat-cwl/cache/ca8d89e0b4b1c166a06924f8a8218b42/result.ms/table.f2
lrwxrwxrwx  1 gijs gijs   85 Jan 30 11:30 table.lock -> /home/gijs/vermeerkat-cwl/cache/ca8d89e0b4b1c166a06924f8a8218b42/result.ms/table.lock

[tetron]

The writable flag means in this context is that the Directory is writable (you can add/delete/rename files, as opposed to a read-only directory) but that doesn't apply recursively to the file contents.

You should be able to use a Javascript expression in the entry field to return a Directory object with the "writable" flag set for each file.

[gijzelaerr]

ok I'll try. But that seems to be quite a workaround hack to try to accomplish something quite simple? We are going to be using this quite often. Lets chat in the chat channel about extending the standard.

[mr-c]

I agree that a shortcut for a recursively writable input directory is useful and a worthy addition to the spec 2017-02-02 9:30 GMT+01:00 Gijs Molenaar <notifications@github.com>:

…

ok I'll try. But that seems to be quite a workaround hack to try to accomplish something quite simple? We are going to be using this quite often. Lets chat in the chat channel about extending the standard. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#282 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABROCA9ojHty4K9-lNcRJw4UOozkTMS8ks5rYZQzgaJpZM4LxM88> .

-- Michael R. Crusoe Community Engineer & Co-founder Common Workflow Language project <http://www.commonwl.org/> https://impactstory.org/u/0000-0002-2961-9670 michael.crusoe@gmail.com +1 480 627 9108 +32 2 808 25 58

[gijzelaerr]

would adding a recursivewritable: true option make sense? Or fullywritable?

[mr-c]

My vote would be for recursiveWritable: true

[mr-c]

A couple points:

writable is only a property on Dirent objects in CWL 1.0, and not anywhere else

But the spec says that when the InitialWorkDirRequirement listing is an Expression it can only return Files and/or Directorys, not Dirents.

However, cwltool does accept and process correctly an Expression for listing that returns an array containing a Dirent

But playing with this beyond-spec behavior I am not seeing a way to specify writable: true recursively for an entire directory. You can't put a Dirent object inside a synthetic Directory object, for example.

bonus: even with writable: false, cwltool doesn't make directories read-only, one can delete files from them

To summarize: We have a clear user story for marking an entire Directory tree as writable. One way forward is to issue a clarification that writable: true for a Directory is recursive and then to update the implementations to match.

[mr-c]

@tetron while tracing the code, I came across some strangness: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/pathmapper.py#L153

[mr-c]

@gijzelaerr wrong button, sorry!

[gijzelaerr]

I guess this issue can be closed? it looks working now for the new release.