git-odb depends on prodash through git-features

[joshtriplett]

It should be possible to pull in a lower-level crate like git-odb without pulling in prodash, or similar high-level output crates.

(This is not in any way urgent, but it'd be nice to reduce the dependencies of applications building on git-odb and similar.)

[Byron]

Thanks for letting me know - some dependency is required as progress is handled through prodash, but I am sure the prodash crate can and should be structured to avoid pulling in more than it needs to.

I will take a look soonish.

[Byron]

Exploring this with cargo +nightly build -Z timings I get the following:

➜  git-odb git:(main) ✗ cargo +nightly build -Z timings  --release
   Compiling libc v0.2.91
   Compiling cfg-if v1.0.0
   Compiling memchr v2.3.4
   Compiling getrandom v0.2.2
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling cc v1.0.67
   Compiling radium v0.5.3
   Completed cfg-if v1.0.0 in 0.1s
   Compiling autocfg v1.0.1
   Completed unicode-xid v0.2.1 in 0.2s
   Compiling build_const v0.2.1
   Completed build_const v0.2.1 in 0.2s
   Compiling pkg-config v0.3.19
   Completed autocfg v1.0.1 in 0.3s
   Compiling quick-error v2.0.0
   Completed radium v0.5.3 custom-build in 0.5s
   Compiling version_check v0.9.3
   Completed libc v0.2.91 custom-build in 0.5s
   Compiling syn v1.0.64
   Completed memchr v2.3.4 custom-build in 0.5s
   Compiling hex v0.4.3
   Completed proc-macro2 v1.0.24 custom-build in 0.5s
   Compiling funty v1.1.0
   Completed getrandom v0.2.2 custom-build in 0.5s
   Compiling tap v1.0.1
   Completed quick-error v2.0.0 in 0.1s
   Compiling smallvec v1.6.1
   Completed tap v1.0.1 in 0.1s
   Compiling wyz v0.2.0
   Completed wyz v0.2.0 in 0.1s
   Compiling ppv-lite86 v0.2.10
   Completed smallvec v1.6.1 in 0.2s
   Compiling crc32fast v1.2.1
   Completed hex v0.4.3 in 0.3s
   Compiling same-file v1.0.6
   Completed syn v1.0.64 custom-build in 0.3s
   Compiling human_format v1.0.3
   Completed pkg-config v0.3.19 in 0.5s
   Compiling scopeguard v1.1.0
   Completed version_check v0.9.3 in 0.4s
   Compiling bytesize v1.0.1
   Completed same-file v1.0.6 in 0.2s
   Compiling sha1 v0.6.0
   Completed cc v1.0.67 in 1.0s
   Compiling arrayvec v0.5.2
   Completed scopeguard v1.1.0 in 0.1s
   Compiling remove_dir_all v0.5.3
   Completed ppv-lite86 v0.2.10 in 0.3s
   Compiling itoa v0.4.7
   Completed bytesize v1.0.1 in 0.2s
   Compiling byteorder v1.4.3
   Completed crc32fast v1.2.1 custom-build in 0.3s
   Compiling instant v0.1.9
   Completed remove_dir_all v0.5.3 in 0.1s
   Compiling crc v1.8.1
   Completed itoa v0.4.7 in 0.2s
   Compiling cmake v0.1.45
   Completed instant v0.1.9 in 0.1s
   Compiling num-traits v0.2.14
   Completed arrayvec v0.5.2 in 0.3s
   Completed human_format v1.0.3 in 0.5s
   Completed byteorder v1.4.3 in 0.3s
   Completed funty v1.1.0 in 0.9s
   Completed sha1 v0.6.0 in 0.4s
   Completed crc v1.8.1 custom-build in 0.3s
   Completed num-traits v0.2.14 custom-build in 0.2s
   Compiling walkdir v2.3.2
   Completed cmake v0.1.45 in 0.3s
   Compiling nom v6.1.2
   Completed nom v6.1.2 custom-build in 0.1s
   Compiling lock_api v0.4.2
   Completed lock_api v0.4.2 in 0.1s
   Compiling prodash v12.0.2
   Completed walkdir v2.3.2 in 0.4s
   Completed radium v0.5.3 custom-build (run) in 0.6s
   Compiling uluru v1.1.1
   Completed uluru v1.1.1 in 0.1s
   Completed prodash v12.0.2 in 0.6s
   Completed libc v0.2.91 custom-build (run) in 1.0s
   Compiling libz-sys v1.1.2
   Completed libz-sys v1.1.2 custom-build in 0.2s
   Completed memchr v2.3.4 custom-build (run) in 1.4s
   Completed radium v0.5.3 in 0.2s
   Completed libc v0.2.91 in 0.2s
   Completed proc-macro2 v1.0.24 custom-build (run) in 1.9s
   Completed memchr v2.3.4 in 0.1s
   Compiling bitvec v0.19.5
   Completed getrandom v0.2.2 custom-build (run) in 2.2s
   Compiling parking_lot_core v0.8.3
   Completed parking_lot_core v0.8.3 in 0.2s
   Compiling filebuffer v0.4.0
   Completed filebuffer v0.4.0 in 0.1s
   Completed syn v1.0.64 custom-build (run) in 2.6s
   Compiling bstr v0.2.15
   Completed proc-macro2 v1.0.24 in 0.4s
   Completed crc32fast v1.2.1 custom-build (run) in 2.5s
   Compiling parking_lot v0.11.1
   Completed bitvec v0.19.5 in 1.0s
   Compiling quote v1.0.9
   Completed bstr v0.2.15 in 0.5s
   Compiling git-hash v0.1.2 (/Users/byron/dev/github.com/Byron/gitoxide/git-hash)
   Completed getrandom v0.2.2 in 0.2s
   Compiling git-ref v0.4.1 (/Users/byron/dev/github.com/Byron/gitoxide/git-ref)
   Completed quote v1.0.9 in 0.2s
   Completed crc v1.8.1 custom-build (run) in 2.7s
   Compiling rand_core v0.6.2
   Completed git-ref v0.4.1 in 0.2s
   Completed git-hash v0.1.2 in 0.2s
   Completed crc32fast v1.2.1 in 0.2s
   Compiling git-features v0.11.0 (/Users/byron/dev/github.com/Byron/gitoxide/git-features)
   Completed crc v1.8.1 in 0.2s
   Compiling rand_chacha v0.3.0
   Completed rand_core v0.6.2 in 0.2s
   Completed parking_lot v0.11.1 in 0.6s
   Completed git-features v0.11.0 in 0.1s
   Compiling rand v0.8.3
   Completed rand_chacha v0.3.0 in 0.2s
   Completed num-traits v0.2.14 custom-build (run) in 3.0s
   Compiling tempfile v3.2.0
   Completed nom v6.1.2 custom-build (run) in 3.0s
   Completed rand v0.8.3 in 0.8s
   Compiling btoi v0.4.2
   Completed num-traits v0.2.14 in 0.6s
   Completed btoi v0.4.2 in 0.1s
   Completed tempfile v3.2.0 in 0.6s
   Compiling git-object v0.6.0 (/Users/byron/dev/github.com/Byron/gitoxide/git-object)
   Completed syn v1.0.64 in 1.9s
   Compiling thiserror-impl v1.0.24
   Completed nom v6.1.2 in 1.2s
   Completed thiserror-impl v1.0.24 in 0.6s
   Compiling thiserror v1.0.24
   Completed thiserror v1.0.24 in 0.6s
   Completed git-object v0.6.0 in 1.3s
   Completed libz-sys v1.1.2 custom-build (run) in 5.4s
   Compiling flate2 v1.0.20
   Completed libz-sys v1.1.2 in 0.1s
   Compiling git-odb v0.9.0 (/Users/byron/dev/github.com/Byron/gitoxide/git-odb)
   Completed flate2 v1.0.20 in 0.3s
   Completed git-odb v0.9.0 in 1.7s
      Timing report saved to /Users/byron/dev/github.com/Byron/gitoxide/git-odb/cargo-timing-20210329T052559Z.html
    Finished release [optimized] target(s) in 11.17s

Prodash and its dependencies consume 0.6s + 0.2s + 0.5s for a 1.3s total. Shaving ~10% off the total build time is not a huge price but worth a look.

Here is how prodash is used:

git-odb git:(main) ✗ rg progress::
src/pack/index/traverse/reduce.rs
2:use git_features::{interrupt::is_triggered, parallel, progress::Progress};

src/pack/index/traverse/indexed.rs
10:    progress::Progress,

src/pack/index/traverse/mod.rs
4:    progress::{self, Progress},
62:            &mut progress::DoOrDiscard<<<P as Progress>::SubProgress as Progress>::SubProgress>,
65:        let progress = progress::DoOrDiscard::from(progress);

src/pack/index/traverse/lookup.rs
6:    progress::{self, unit, Progress},
59:                    p.init(Some(self.num_objects() as usize), progress::count("objects"));

src/pack/index/verify.rs
2:use git_features::progress::{self, Progress};
127:        let mut root = progress::DoOrDiscard::from(progress);

src/pack/data/verify.rs
2:use git_features::progress::Progress;
27:    /// Note that if no `progress` is desired, one can pass [`git_features::progress::Discard`].

src/pack/bundle/write/mod.rs
4:use git_features::{interrupt, progress, progress::Progress};
22:    /// `progress` provides detailed progress information which can be discarded with [`git_features::progress::Discard`].
31:        read_progress.init(None, progress::bytes());
32:        let pack = progress::Read {
34:            progress: progress::ThroughputOnDrop::new(read_progress),
77:        read_progress.init(pack_size.map(|s| s as usize), progress::bytes());
78:        let pack = progress::Read {
80:            progress: progress::ThroughputOnDrop::new(read_progress),

src/pack/index/write/mod.rs
5:use git_features::progress::{self, Progress};
83:        root_progress.init(Some(4), progress::steps());
85:        objects_progress.init(entries.size_hint().1, progress::count("objects"));
87:        decompressed_progress.init(None, progress::bytes());
205:            progress::count("objects").expect("unit always set"),

src/pack/index/util.rs
2:use git_features::progress::{self, Progress};
9:    progress.init(Some(idx.num_objects as usize), progress::count("entries"));

src/pack/tree/traverse/resolve.rs
5:use git_features::progress::{unit, Progress};

src/pack/index/write/encode.rs
6:use git_features::progress::{self, Progress};
42:    progress.init(Some(4), progress::steps());
125:        progress::bytes().expect("unit always set"),

src/pack/tree/traverse/mod.rs
10:    progress::{self, Progress},
128:            .init(Some(num_objects as usize), progress::count("objects"));
129:        size_progress.init(None, progress::bytes());

tests/pack/file.rs
25:        assert_eq!(p.verify_checksum(progress::Discard)?, p.checksum());

src/pack/tree/from_offsets.rs
4:    progress::{self, Progress},
59:            progress.init(Some(num_objects), progress::count("objects"));

tests/pack/tree.rs
25:                git_features::progress::Discard,

tests/pack/index.rs
124:                progress::Discard,
297:                        progress::Discard.into()
393:            idx.verify_integrity(None::<(_, _, _, fn() -> cache::Noop)>, None, progress::Discard.into())

tests/pack/bundle.rs
146:            progress::Discard,

Even though my intuition was that maybe it won't use both the bytesize or even better human_format crates, it in fact does use both via progress::bytes() and progress::count()

prodash itself is already heavily feature toggled, and it won't provide more than the given unit implementations and the Progress trait itself. Sure, there is also the 'messages' module, but it has less than 100 lines of code and not compiling it probably won't save much.

In the light of this information, do you have other ideas on how to shave some time off? Personally I think that human_format costs quite a lot to compile for what I thought it was (it's just 226 lines of code and absolutely nothing fancy).

Edit: In any case, it's quite interesting to see that merely having progress currently costs about a whole second of a programs build time - not super awesome but bearable.

[joshtriplett]

I didn't realize that prodash was also providing the infrastructure for progress handling, not just the UI. The README of the prodash crate says "dashboard for displaying progress of concurrent applications" at the top. But, digging further, I see that prodash has features to control all the renderers. Given that, I don't think any change to gitoxide is needed.

Thanks for the detailed explanation and analysis!

[Byron]

Thanks for bringing it up, there is a little bit of potential to save a dependency possibly with other crates.

In any case, I hope I made its description a bit clearer.