Skip to content

[perf] Skip cache check if store path exists locally #2042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 9, 2024
Merged

[perf] Skip cache check if store path exists locally #2042

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions internal/devpkg/narinfo_cache.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@ import (
"fmt"
"io"
"net/http"
"os"
"sync"
"time"

"github.com/pkg/errors"
"go.jetpack.io/devbox/internal/boxcli/featureflag"
"go.jetpack.io/devbox/internal/debug"
"go.jetpack.io/devbox/internal/lock"
"go.jetpack.io/devbox/internal/nix"
"golang.org/x/sync/errgroup"
Expand Down Expand Up @@ -51,6 +53,7 @@ func (p *Package) IsInBinaryCache() (bool, error) {
// package in the list, and caches the result.
// Callers of IsInBinaryCache may call this function first as a perf-optimization.
func FillNarInfoCache(ctx context.Context, packages ...*Package) error {
defer debug.FunctionTimer().End()
if !featureflag.RemoveNixpkgs.Enabled() {
return nil
}
Expand Down Expand Up @@ -145,6 +148,11 @@ func (p *Package) fetchNarInfoStatus(outputName string) (bool, error) {

outputInCache := map[string]bool{} // key = output name, value = in cache
for _, output := range outputs {
// if store path exists locally, then it is equivalent to being in the cache.
Copy link
Collaborator

@savil savil May 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is correct, assuming the nix order for resolving a store-path is:

  1. check local /nix/store to see if already built
  2. check binary cache(s) configured to see if binaries are present
  3. build derivation locally

I say assuming because looking at the docs for builtins.fetchClosure it doesn't say that it will check local nix-store and skip the binary cache if its present:
https://nixos.org/manual/nix/stable/language/builtins.html#builtins-fetchClosure

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume it works this way, because otherwise we would be re-downloading everything every time the flake is evaluated. Currently we build everything first (which stores it int the nix store) and then we evaluate the flake using print-dev-env.

Also, see this note:

fetchClosure is similar to builtins.storePath in that it allows you to use a previously built store path in a Nix expression. However, fetchClosure is more reproducible because it specifies a binary cache from which the path can be fetched

builtins.storePath always checks the local store first before using substituters, so I assume fetchClosure does the same before trying to fetch from remote.

cc: @gcurtis let us know if this is your understanding as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's also my understanding.

if _, err := os.Stat(output.Path); err == nil {
outputInCache[output.Name] = true
continue
}
pathParts := nix.NewStorePathParts(output.Path)
reqURL := BinaryCache + "/" + pathParts.Hash + ".narinfo"
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
Expand Down