Add a way to run bob with a specific list of updated resources

[astrochili]

The build time with bob is longer than in Defold Editor.

This is noticeable when using the Defold Kit VS Code extension, which uses bob to build and run the game for debugging. But in this case, when using Defold Kit most often you don't need to rebuild all the resources, only the modified ones, like it works in Defold Editor.

Tracking of changed files is possible in VS Code, there is an API for that. So adding tracking to Defold Kit should not be difficult.

This feature request is about adding some API for bob to rebuild only the list of changed resources and not waste time on unchanged ones. I realise it's low priority, but added this issue to have it.

Related issue — astrochili/vscode-defold#33.

[swajj]

I would love to see bob incremental builds- I also build using bob, but mostly via shell scripting. I would really love if bob's output included a cache of file modification times or some other data so that I could run bob incrementally as long as that cache persisted. I maintain multiple concurrent build directories (client / server) so this would need to be cached by build. I see there's already a _BobBuildState_ file in there- is this something that's already being tracked perhaps?

[ekharkunov]

Take a look at https://defold.com/manuals/caching-assets/. Maybe it will be helpful.

[britzl]

Take a look at https://defold.com/manuals/caching-assets/. Maybe it will be helpful.

Yep, adding a local cache will surely help a bit. But I suppose one problem with bob compare to the editor is that the editor is loaded, ready and has the full project state in memory, which makes it really fast to rebuild changed assets. Bob on the other hand need to start a java vm, load the classes of bob.jar, load the entire project and the build it. I would like to point out that bob will not rebuild something that is already built, but it does ofc have to do some processing (read file and measure checksums) to figure out if it has to be rebuilt or not.

[astrochili]

Yep, adding a local cache will surely help a bit.

Is this the equivalent of running the build command without the distclean command? I thought there was a cache in the /build/ folder, which is reused when you run bob build again.

[astrochili]

May be we need measurements of some large project to see what takes the longest.

I don't have huge projects. My small ones with a dozen collections, several atlases and half a hundred lua scripts take 5 to 15 seconds to build second time, which is acceptable to me. But in the editor, yes, it is faster, takes about 2-3 seconds.

@swajj As far as I know you have not a small project. Maybe you could share the build log measurements?

[JCash]

Bob already supports incremental builds. It does so by using a cache.

Are you sure you're not calling it with the clean command?
If doing so, it will rebuild all content.

Also note that bob was/is designed to be run once, not to be run as a server (like the editor), so you'll always pay for the startup time (compare with the loading of a project in the editor).

I'm not sure what we can do to speed up the initial loading of the project when starting bob.

[AGulev]

Also, the process is more complex than simply checking if a file has changed to decide on a rebuild. As Björn mentioned:

but it does of course have to do some processing (read the file and measure checksums) to figure out if it has to be rebuilt or not.

However, it involves more than just reading the file and getting its checksum. We run the create stage for each builder (see @Override public Task<Void> create(IResource input) in each *Builder.java). This is crucial because the system does not operate on a one input -> one output basis. Some inputs produce multiple outputs, and some outputs require multiple inputs. Additionally, there are dependencies, where one output can be utilized as an input for another resource tree node.

The editor maintains the dependency tree in memory and always knows exactly what needs to be rebuilt at any point in time. Bob, on the other hand, cannot do this and must read this tree from the project files all the time (that's what the create() function in each builder does). That's what takes most of the time. You can check it yourself by producing html build report.

[astrochili]

Are you sure you're not calling it with the clean command?

Sure, there is a difference. I don't have large projects, so maybe my measurements aren't representative, but I tried to record 4 logs and separate in them intervals that either take appreciable time or are very different between the first and second builds.

• catsday_clean.log — a clean first build of catsday.
• catsday_second.log — a second build of catsday without cleaning.
• walker_clean.log — a clean first build of walker.
• walker_second.log — a second build of walker without cleaning.

Despite saving on matches (I'm personally happy with the current build time anyway), what can we see here that's interesting? The first number is clean build and the second is an incremental build.

Total for catsday — 7.8 vs 7.4 seconds 🤷‍♂️.

• 0% to 99% — 4.1 vs 1.3 seconds ✅.
• 99% to 100% — 1.0 vs 4.5 seconds 🤷‍♂️.
• other — 2.7 vs 1.6 seconds ✅.

Total for walker — 3.8 vs 1.7 seconds ✅.

• 0% to 99% — 2.2 vs 0.6 seconds ✅.
• 99% to 100% — 0.6 vs 0.1 seconds ✅.
• other — 1.0 vs 1.0 seconds ✅.

So, I don't know, if 0% to 99% is what is cached and the rest is environment initialization, then there's really nothing to speed up.

[JCash]

Yeah, not sure what things are processed in the actual last bit (99% to 100%), and it seems worth investigating.
So, same end goal, but a slightly different task.