make() with multiple targets very slow, seems to do too much checking

[krlmlr]

I don't have an all target, instead I run make(list_targets()). On a mid-size project with many interdependent targets (~4 GB overall file size) it takes about 4 minutes just to detect what needs to be done, even if the project is up to date. Checking a single target (with many indirect dependencies) is a lot faster.

For this particular case, I'd be fine if a missing all targets mean that by default all targets are built.

[richfitz]

yowch. There have been previous reports of slowness but I don't have anything to benchmark with. If you can profile your problem (or share it) it would be nice to get to the bottom of it.

I'm not surprised that there's too much checking though - the original version was written for correctness rather than speed and I never got around to actually trying to speed it up. I'd hazard a guess that the issue is that objects are being loaded into R when they can just be skipped over...

[krlmlr]

I'll see if I can whip up a reprex.

[krlmlr]

I think the main cause here is the hashing of file targets. Using check: exists should speed up things, but I'll keep this open because I think ideally targets should be checked only once per remake run.

[richfitz]

Right - so within a single run of remake there's no need to rehash them. That seems totally worth implementing.

We could probably go through further; files that are created by remake should never need to be hashed because we know the hash when we write it to the DB.

As a third related thing; hashing large files is big and expensive, but I wonder if it would not suffice in many situations to hash the first (or last 'n') bytes along with the size. I believe that reading the file (rather than the hashing itself) is the bottleneck here.

[richfitz]

Can you try the hash_files branch and see if this helps at all?

[richfitz]

Hmm, not sure why appveyor didn't like that - might be something in the mocking...

[krlmlr]

I have commented the PR., have you seen #126? #126 helps for single, but not for repeated make() calls.

[richfitz]

Yeah, that looked good - I just need to merge it :)

(I will get more up to speed on all of this over the next couple of weeks, just taking a bit of time to shift gears between projects)