Copier overrides identical files

[Smidimir]

Is there any reason to override file if copier detects that rendered file is identical? It surprised me, that copier after ensuring that file have the same content, writes it anyway. I made this change that skips identical files.

I wanted to make a pull request with this change, but I couldn't setup tests to check if this change could break anything.

First reason I needed this change, is that some programs (in my case msvc compiler for c++) rely on file properties (like timestamp) to check if there were any changes. So after rendering template where 100% of files have the same content, msvc recompiles the whole project from scratch because every timestamp of every file was updated.

Second reason is speed. In my case I got about 20% speedup for copying template with same result (6 min -> 5 min). I have template with a lot of files.

[yajo]

No particular reason. Just it's "the same". Probably when dealing with updates you'll get some stuff changed anyway due to the smart diff algorithm. But it's not a feature but a side effect. Anything you can improve, do it please 😃 El mar, 11 abr 2023, 9:23, Smidimir ***@***.***> escribió:

…

Is there any reason to override file if copier detects that rendered file is identical? It surprised me, that copier after ensuring that file have the same content, writes it anyway. I made this change <cecc0e6> that skips identical files. I wanted to make a pull request with this change, but I couldn't setup tests to check if this change could break anything. First reason I needed this change, is that some programs (in my case msvc compiler for c++) rely on file properties (like timestamp) to check if there were any changes. So after rendering template where 100% of files have the same content, msvc recompiles the whole project from scratch because every timestamp of every file was updated. Second reason is speed. In my case I got about 20% speedup for copying template with same result (6 min -> 5 min). I have template with a lot of files. — Reply to this email directly, view it on GitHub <#1092>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHNXDORWGXH7QIJB4JYAXLXAUIJNANCNFSM6AAAAAAWZ56WWQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Smidimir]

Thanks for clarification. In that case, I'll try to submit pull request later. Once I figure out how to run tests properly 😅. I work on windows and tried to setup them in WSL. But when I ran them there were a lot of failed tests. I tried to run tests with and without changes. So, I just assumed that my setup was wrong. But I also didn't dig deeper into the problem.

[yajo]

After reviewing it in more depth, it indeed looks like a nice improvement. You can submit the PR and let tests run on CI, in case it helps.

[kefirbandi]

Hi All, I implemented the change and run the CI workflow: https://github.com/kefirbandi/copier/actions/runs/8578301208
It has a few failures, all related to missing codecov token. Are these failures acceptable? Shall I proceed with the PR?

[yajo]

that's testing your master branch. The tests for the other Branch will only run when you open the PR. The milling codecov token is due to the fact that you're running tests on your own fork.

cecc0e6 is still missing a test to be merged. the test should probably check the modification time to make sure that the file doesn't change.

My suggestion is that you open the PR and we discuss it there. In the PR CI we'll see if it works. Then you can add the test and we can merge.

[kefirbandi]

I submitted a new PR, which has the test included: #1593

[kefirbandi]

I ran into the exact same problem. But in my case it is not only speed, but the process fails randomly with 30% chance on the network drive. (That is 30% on each file. If I have 10 files, a failure is guaranteed.)

This really is a single line of code to change. I guess it wouldn't take more than 10 minutes for the developers to fix and run the tests. But for us, mortals, checking out the code, struggling with the failed tests, it could take hours before we give up.

I would be glad if you would consider implementing this change.

[yajo]

The machinery involved in updates is very complex. Most changes are done by Git and not by Copier, and we need to do them to check if the final state of the file is the same as the previous state of the file + all the template updates that happened in between.

Fixing an issue without being able to reproduce it isn't easy also. This is because I don't have a failing network drive.

Have you considered fixing your network or drive? A 30% failure chance is very high! And Copier isn't guilty of hardware or network misconfigurations... There are sync systems that cache changes locally to replicate them remotely when the network is fine. Maybe you need a different tech stack there?

[kefirbandi]

Hi yajo, thanks for the reply.
I am not in charge of the tech stack, so I am not in the position to change it.
I now locally implemented the change, which was suggested by Smidimir above and it works perfectly.

So if you could please consider that pull requests, run the tests, and if nothing brakes then I think it would make life easier for many people.

Because I have some issues with the network drive, but I can't think of a scenario where people would want to overwrite identical files.

thank you