bug in (extra) output to different directory and %O

[FreddieChopin]

If I have a Tupfile like this one:

: |> echo first > %o && echo second > %o.2 |> out/file.txt | %O.txt.2

It fails with following message:

d:\1\tup_dir>tup
[ tup ] [0.001s] Scanning filesystem...
[ tup ] [0.114s] Reading in new environment variables...
[ tup ] [0.114s] Parsing Tupfiles...
 1) [0.001s] .
 [ ] 100%
[ tup ] [0.117s] Deleting files...
 1) rm: out\file.txt.2
 [ ] 100%
[ tup ] [0.120s] Generating .gitignore files...
[ tup ] [0.239s] Executing Commands...
* 1) echo first > out/file.txt && echo second > out/file.txt.2
 *** tup errors ***
tup error: File 'd:\1\tup_dir\out/file.txt.2' was written to, but is not in .tup
/db. You probably should specify it as an output
 -- Delete: d:\1\tup_dir\out/file.txt.2
tup error: Expected to write to file 'out/file.txt.2' from cmd 25 but didn't
 *** Command ID=25 ran successfully, but tup failed to save the dependencies.
 [ ] 100%
 *** tup: 1 job failed.

As you notice tup somehow does not like the output to the file it expects...

This works if the output goes to the same directory:

: |> echo first > %o && echo second > %o.2 |> file.txt | %O.txt.2

It also works if I state the "extra output" explicitly (so if I replace "%O" with "out/file"):

: |> echo first > %o && echo second > %o.2 |> out/file.txt | out/file.txt.2

Interesting fact - %O does not care about preceding it with "out/" - in both cases the output is the same (same error).

[gittup]

On Wed, Nov 20, 2013 at 8:12 AM, freddiechopin notifications@github.comwrote:

If I have a Tupfile like this one:

: |> echo first > %o && echo second > %o.2 |> out/file.txt | %O.txt.2

It fails with following message:

d:\1\tup_dir>tup
[ tup ] [0.001s] Scanning filesystem...
[ tup ] [0.114s] Reading in new environment variables...
[ tup ] [0.114s] Parsing Tupfiles...

1. [0.001s] .
   [ ] 100%
   [ tup ] [0.117s] Deleting files...
2. rm: out\file.txt.2
   [ ] 100%
   [ tup ] [0.120s] Generating .gitignore files...
   [ tup ] [0.239s] Executing Commands...

• 1. echo first > out/file.txt && echo second > out/file.txt.2
     *** tup errors ***
     tup error: File 'd:\1\tup_dir\out/file.txt.2' was written to, but is not in .tup
     /db. You probably should specify it as an output
     -- Delete: d:\1\tup_dir\out/file.txt.2
     tup error: Expected to write to file 'out/file.txt.2' from cmd 25 but didn't
     *** Command ID=25 ran successfully, but tup failed to save the dependencies.
     [ ] 100%
     *** tup: 1 job failed.

The path length calculation on %O was incorrect - it was using the length
of the base filename and then copying a part of the full path. I believe
this is now fixed (t4160) - let me know if it doesn't fix your actual use
case.

Thanks for the report!
-Mike

[FreddieChopin]

Hmm... I thought I have given a comment yesterday, but it seems I did not (or it got lost?)...

So, the base issue is indeed solved. There are - however - some strange corner cases, which don't work as expected and give very strange and cryptic messages.

1. A slight modification of the original example (extra "out/" before second output):

   : |> echo first > %o && echo second > %o.2 |> out/file.txt | out/%O.txt.2

Gives a really cryptic and confusing error - the file that was written is wrong, but it is the file that it expects to be written to...

d:\1\tup_dir>tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.001s] Reading in new environment variables...
[ tup ] [0.002s] No Tupfiles to parse.
[ tup ] [0.002s] No files to delete.
[ tup ] [0.003s] Executing Commands...
* 1) echo first > out/file.txt && echo second > out/file.txt.2
 *** tup errors ***
tup error: File 'd:\1\tup_dir\out/file.txt.2' was written to, but is not in .tup
/db. You probably should specify it as an output
 -- Delete: d:\1\tup_dir\out/file.txt.2
tup error: Expected to write to file 'out/file.txt.2' from cmd 38 but didn't
 *** Command ID=38 ran successfully, but tup failed to save the dependencies.
 [ ] 100%
 *** tup: 1 job failed.

1. Second output to a different directory:

   : |> echo first > %o && echo second > another%o.2 |> out\file.txt | another%O.txt.2

At first system's shell complains about the missing directory, as "another" is created, but "another\out" - not.

d:\1\tup_dir>tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.100s] Reading in new environment variables...
[ tup ] [0.101s] Parsing Tupfiles...
 1) [0.000s] .
 [ ] 100%
[ tup ] [0.104s] No files to delete.
[ tup ] [0.104s] Generating .gitignore files...
[ tup ] [0.208s] Executing Commands...
* 1) echo first > out/file.txt && echo second > another\out/file.txt.2
System nie może odnaleźć określonej ścieżki.
 *** tup errors ***
 *** Command ID=38 failed with return value 1
tup error: Expected to write to file 'out/file.txt.2' from cmd 38 but didn't
 *** Additionally, command 38 failed to process input dependencies. These should
 probably be fixed before addressing the command failure.
 [ ] 100%
 *** tup: 1 job failed.

If I just create this directory manually:

d:\1\tup_dir>tup
[ tup ] [0.001s] Scanning filesystem...
[ tup ] [0.095s] Reading in new environment variables...
[ tup ] [0.096s] No Tupfiles to parse.
[ tup ] [0.096s] No files to delete.
[ tup ] [0.097s] Executing Commands...
* 1) echo first > out/file.txt && echo second > another\out/file.txt.2
 *** tup errors ***
tup error: Unspecified output files - A command is writing to files that you did
n't specify in the Tupfile. You should add them so tup knows what to expect.
 -- Delete: file.txt.2 at dir 43
tup error: Expected to write to file 'out/file.txt.2' from cmd 38 but didn't
 *** Command ID=38 ran successfully, but tup failed to save the dependencies.
 [ ] 100%
 *** tup: 1 job failed.

[gittup]

On Fri, Dec 6, 2013 at 5:44 PM, freddiechopin notifications@github.comwrote:

Hmm... I thought I have given a comment yesterday, but it seems I did not
(or it got lost?)...

So, the base issue is indeed solved. There are - however - some strange
corner cases, which don't work as expected and give very strange and
cryptic messages.

A slight modification of the original example (extra "out/" before
second output):

: |> echo first > %o && echo second > %o.2 |> out/file.txt |
out/%O.txt.2

What is the expected behavior here? I guess you're expecting %O to trim
both 'out/' and '.txt' so it would just be 'file', right? Ie: it should
generate "out/file.txt" and "out/file.txt.2", not "out/out/file.txt.2"?

Gives a really cryptic and confusing error - the file that was written is
wrong, but it is the file that it expects to be written to...

Looking at the graph that's generated, tup is rather oddly expecting a
filename called "out/file.txt.2" to be created in the directory "out/" - in
other words, it's looking for a filename with a '/' character in it.
Obviously a bug :). I've filed #162

-Mike

[FreddieChopin]

What is the expected behavior here? I guess you're expecting %O to trim
both 'out/' and '.txt' so it would just be 'file', right? Ie: it should
generate "out/file.txt" and "out/file.txt.2", not "out/out/file.txt.2"?

Actually I think that the second file generated should be out/out/file.txt.2 - after all %O is the full name of the first output, including the folder. The only problem here is that this particular case does not work and the error it gives is not very helpful (;

Regards,
FCh

[gittup]

On Fri, Jan 3, 2014 at 2:01 AM, freddiechopin notifications@github.comwrote:

What is the expected behavior here? I guess you're expecting %O to trim
both 'out/' and '.txt' so it would just be 'file', right? Ie: it should
generate "out/file.txt" and "out/file.txt.2", not "out/out/file.txt.2"?

Actually I think that the second file generated should be
out/out/file.txt.2 - after all %O is the full name of the first output,
including the folder. The only problem here is that this particular case
does not work and the error it gives is not very helpful (;

I think this one is now fixed too. Let me know if you stumble on any
others!

-Mike

[FreddieChopin]

Yes, it's better now, but the error message is a bit misleading still. The "wrong" tupfile - as presented above - gives this:

d:\1\tup_dir>tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.107s] Reading in new environment variables...
[ tup ] [0.108s] Parsing Tupfiles...
 1) [0.001s] .
 [ ] 100%
[ tup ] [0.110s] No files to delete.
[ tup ] [0.111s] Generating .gitignore files...
[ tup ] [0.200s] Executing Commands...
* 1) echo first > out\file.txt && echo second > out\file.txt.2
 *** tup errors ***
tup error: File 'd:\1\tup_dir\out\file.txt.2' was written to, but is not in .tup
/db. You probably should specify it as an output
 -- Delete: d:\1\tup_dir\out\file.txt.2
tup error: Expected to write to file 'file.txt.2' from cmd 21 but didn't
 *** Command ID=21 ran successfully, but tup failed to save the dependencies.
 [ ] 100%
 *** tup: 1 job failed.

It would be great if it could complain about not writing to out/out/file.txt.2 instead of just "file.txt.2".

The fixed file (with another "out/" in the command) works fine.

Thx!

[gittup]

On Tue, Jan 7, 2014 at 4:00 PM, freddiechopin notifications@github.comwrote:

Yes, it's better now, but the error message is a bit misleading still. The
"wrong" tupfile - as presented above - gives this:

d:\1\tup_dir>tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.107s] Reading in new environment variables...
[ tup ] [0.108s] Parsing Tupfiles...

1. [0.001s] .
   [ ] 100%
   [ tup ] [0.110s] No files to delete.
   [ tup ] [0.111s] Generating .gitignore files...
   [ tup ] [0.200s] Executing Commands...

• 1. echo first > out\file.txt && echo second > out\file.txt.2
     *** tup errors ***
     tup error: File 'd:\1\tup_dir\out\file.txt.2' was written to, but is not in .tup
     /db. You probably should specify it as an output
     -- Delete: d:\1\tup_dir\out\file.txt.2
     tup error: Expected to write to file 'file.txt.2' from cmd 21 but didn't
     *** Command ID=21 ran successfully, but tup failed to save the dependencies.
     [ ] 100%
     *** tup: 1 job failed.

It would be great if it could complain about not writing to
out/out/file.txt.2 instead of just "file.txt.2".

Oh, good point. This should be fixed now too. Keep 'em coming!

-Mike