Is there a declarative way to not apply substitutions during file copying?

[bombaywalla]

I was working on developing a deps-new template for cryogen. That project contains several files that already contain {{something}} in them (since they are selmer templates) that should not be substituted during copying to a new project. For example, the current cryogen lein template generates files that contain {{name}}.

I guess I could use :data-fn but was looking for something more declarative, if available.

Thanks.

[seancorfield]

It's not documented yet (although it is shown in the Spec for template.edn) but you can specify custom open/close delimiters for the substitution via the :transform structure, on a per-directory basis.

See https://github.com/seancorfield/deps-new/blob/develop/resources/org/corfield/new/template/template.edn#L3 for an example of this in the template template.

Here it is in the Spec: https://github.com/seancorfield/deps-new/blob/develop/src/org/corfield/new.clj#L20-L23

I wanted to give it some time to "bake", with folks using the template template, before I documented it, in case it needed changes.

[bombaywalla]

Thanks. If I use

{:transform
 [["content" "content" {} ["<<" ">>"]]
  ["themes" "themes" {} ["<<" ">>"]]]}

that seems to copy over the text files from the template to the new project as expected (i.e., without any changes).
However, there seems to be a problem with a binary (in this case, the cryogen PNG) file.

In case it helps, here is the output of od -c cryogen.png | head on the template file

0000000  211   P   N   G  \r  \n 032  \n  \0  \0  \0  \r   I   H   D   R
0000020   \0  \0 003 350  \0  \0 001   ^  \b 006  \0  \0  \0 026 257   5
0000040  245  \0  \0  \0 004   s   B   I   T  \b  \b  \b  \b   |  \b   d
0000060  210  \0  \0  \0  \t   p   H   Y   s  \0  \0  \r 327  \0  \0  \r
0000100  327 001   B   ( 233   x  \0  \0  \0 031   t   E   X   t   S   o
0000120    f   t   w   a   r   e  \0   w   w   w   .   i   n   k   s   c
0000140    a   p   e   .   o   r   g 233 356   < 032  \0  \0      \0   I
0000160    D   A   T   x 234 354 335   y 234   d   U   a 366 361 337 251
0000200  352   e   v 006 030 030   V 025 025 020   A 005   9 020 224 001
0000220  001   q   A   d 021   E   b   ܢ  **   ! 232 250 021   M   \   b

And here is the output of the same command on the copied over file.

0000000    �  **  **   P   N   G  \r  \n 032  \n  \0  \0  \0  \r   I   H
0000020    D   R  \0  \0 003   �  **  **  \0  \0 001   ^  \b 006  \0  \0
0000040   \0 026   �  **  **   5   �  **  **  \0  \0  \0 004   s   B   I
0000060    T  \b  \b  \b  \b   |  \b   d   �  **  **  \0  \0  \0  \t   p
0000100    H   Y   s  \0  \0  \r   �  **  **  \0  \0  \r   �  **  ** 001
0000120    B   (   �  **  **   x  \0  \0  \0 031   t   E   X   t   S   o
0000140    f   t   w   a   r   e  \0   w   w   w   .   i   n   k   s   c
0000160    a   p   e   .   o   r   g   �  **  **   �  **  **   < 032  \0
0000200   \0      \0   I   D   A   T   x   �  **  **   �  **  **   �  **
0000220   **   y   �  **  **   d   U   a   �  **  **   �  **  ** 337 251

This is using deps-new v0.3.3 installed as a Clojure tool.

Any idea what I might be doing wrong?

[seancorfield]

deps-new uses tools.build for file copying -- see if you can reproduce the issue with just b/copy-dir and/or b/copy-file. I would have expected those functions to copy binary files without changing them, but I haven't looked into their code really.

[bombaywalla]

Okay. Will try that. Thanks.

[bombaywalla]

I was able to reproduce this (as you suggested) using just b/copy-dir.
Without a :replace parameter, b/copy-dir copies the files over faithfully.
With the :replace parameter set to {"<<name>>" "me"}, it corrupts the binary PNG file.

Filed a question on ask.clojure.org.
https://ask.clojure.org/index.php/11147/tools-build-api-copy-dir-corrupts-binary-files-replace-used

[seancorfield]

I've been thinking about some sort of "raw" copy in deps-new so this kind of forces my hand. I'll give it some thought but I'm leaning toward some sort of :raw indicator to suppress the :replace option, but I need to think about how the DSL for :transform is going to end up looking at this point.

[seancorfield]

Undocumented but you can try it out on the latest SHA on develop -- if you declare a :transform entry that has :raw at the end (in a particular tuple), it won't use :replace and will copy files as-is, which should solve the binary file issue -- although you'll need to have those in a separate :transform tuple.

[seancorfield]

I've updated the docs https://github.com/seancorfield/deps-new/blob/develop/doc/templates.md#suppressing-substitution--binary-files

[bombaywalla]

Thanks. I tried this SHA (and the latest SHA) and I am getting an error:

Dorabs-iMac:Projects dorab$ clojure -Sdeps '{:deps {c/c {:local/root "depscryogen"}}}' -Tnew create :template myorg/depscryogen :name test2/myblog :force true
Execution error (IllegalArgumentException) at org.corfield.new/create (new.clj:63).
Additional data must be non-nil.

Full report at:
/var/folders/lk/mf8f6ghs0zqcw70nmy739khm0000gn/T/clojure-14425458454229985389.edn
Dorabs-iMac:Projects dorab$ clojure -Sdeps '{:deps {c/c {:local/root "depscryogen"}}}' -Tnew create :template myorg/depscryogen :name test2/myblog
Execution error (IllegalArgumentException) at org.corfield.new/create (new.clj:63).
Additional data must be non-nil.

Full report at:
/var/folders/lk/mf8f6ghs0zqcw70nmy739khm0000gn/T/clojure-6014328606993832223.edn

I took a look at the code, and I believe the problem is at Line 62 of new.clj.
(when-not (s/invalid? edn') should be (when (s/invalid? edn').

[seancorfield]

Oh, good grief! Sorry, I tested some of the low-level stuff and totally missed that! Try the latest SHA again.

[bombaywalla]

Thanks. 2b7e324 now works, but results are different from v0.3.3.

The main issue I see is that the files under root are copied over to a folder called root under the <target>, rather than being under the <target> dir itself.

[bombaywalla]

And, it is copying over template.edn as well.

[seancorfield]

Maybe Sunday isn't a good day to do open source work? :) Yeah, this is because I changed the Spec stuff to s/conform it but I forgot to s/conform the literal parameter. Bear with me...

[seancorfield]

OK, try the latest SHA now...

[bombaywalla]

It works! Passes my rudimentary tests for creating the cryogen template.
Thanks

[seancorfield]

Awesome. Finally! Thank you for bearing with me. I'll leave develop alone for a few days and if you don't run into any other problems, I'll cut 0.4.0.

[bombaywalla]

Glad I could help.
I learned a few things during this process, so it was all worthwhile.

[seancorfield]

I see Alex just released tools.build 0.6.0 with :non-filtered-exts as a list of file extensions that shouldn't have replacements applied which kind of makes this work moot -- although having directories copied as-is without replacement is probably useful beyond image files. I'll have to update the docs again (and bump the tools.build dependency).

[seancorfield]

v0.4.0 released with latest tools.build and build-clj -- by default common image types (like png) are no longer substituted and should be copied as binary files, but the :raw directive will still be useful for other binary files and for template files.