file:// custom datasource can't read data when extract is served from repositoryCache (data: null → ZodError → "no-result")

[negtak]

How are you running Renovate?

CLI

Which platform you running Renovate on?

Local

Which version of Renovate are you using?

43.209.4

Please tell us more about your question or problem

Problem?

I update a dependency by reading another file in the repo: a regex custom manager produces the
dep, and a customDatasources entry with a file:// registryUrl reads a JSON file out of the
checkout to get the list of versions.

It works on a cold run but breaks as soon as the extract result is served from
repositoryCache. Same repo, same commit — the only difference is whether the extract cache is
warm. On the warm run the custom datasource gets data: null, which then fails Zod validation
and the dep is reported as not found.

The reason is that the file:// datasource reads from the working tree during lookup, but
the base branch is only checked out on the extract cache-miss path. On a cache hit the checkout
is skipped, so the working tree is still on the default branch and the file:// target
isn't there.

How to reproduce

renovate.json (present on both branches):

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "customDatasources": {
    "local-versions": {
      "transformTemplates": ["$.\"{{{packageName}}}\""]
    }
  },
  "customManagers": [
    {
      "customType": "regex",
      "managerFilePatterns": ["/app.txt$/"],
      "matchStrings": ["# renovate: depName=(?<depName>\\S+)\\s+\\S+=(?<currentValue>\\S+)"],
      "datasourceTemplate": "custom.local-versions",
      "registryUrlTemplate": "file://versions/{{{depName}}}.json"
    }
  ]
}

(Note: the registryUrl templatee {{{depName}}}.{{{packageName}}} is empty at extract time, so it would resolve to versions/.json — that's a separate gotcha.)

app.txt

# renovate: depName=foo
foo=1.0.0

versions/foo.json

{ "foo": { "releases": [{ "version": "1.0.0" }, { "version": "1.1.0" }] } }

Run it twice with the cache enair so the cache survives:

# Run A: cold cache
renovate --platform=github --repository-cache=enabled --base-dir=/tmp/rn sample/<repo>
# Run B: warm cache
renovate --platform=github --repository-cache=enabled --base-dir=/tmp/rn sample/<repo>

Logs

Run A (cold) — works, foo 1.0.0 -> 1.1.0 is proposed:

DEBUG: Setting current branch to develop
...
INFO:  PR created ... chore(deps): update dependency foo to v1.1.0

Run B (warm) — fails:

DEBUG: Cached extract for sha=3cdeabed… is valid and can be used
        (no "Setting current branch" line — checkoutBranch is skipped)
TRACE: Error reading local file
       ENOENT: no such file or directory, open '.../<repo>/versions/foo.json'
TRACE: Custom datasource API fetcher 'json' received data
       "data": null
DEBUG: Response has failed validation
       ZodError: "Invalid input: expected object, received undefined"
DEBUG: Failed to look up custom.local-versions package foo: no-result
WARN:  Package lookup failures

What's going on

The file:// datasource reads the working tree at lookup time
(lib/modules/datasource/custom/index.ts:38-41 → formats/json.ts:11-15, i.e.
readLocalFile() + JSON.parse)

The only place that checks out miss path of extract():

// lib/workers/repository/process/extract-update.ts
  if (
    overwriteCache &&
    isCacheExtractValid(baseBranchSha!, configHash, cachedExtract)
  ) {
    packageFiles = cachedExtract.packageFiles;
    try {
      for (const files of Object.values(packageFiles)) {
        for (const file of files) {
          for (const dep of file.deps) {
            delete dep.updates;
          }
        }
      }
      logger.debug('Deleted cached dep updates');
    } catch (err) {
      logger.info({ err }, 'Error deleting cached dep updates');
    }
  } else {
    await instrument(
      'checkoutBranch',
      async () => await scm.checkoutBranch(baseBranch!),
    );

Lookup runs afterwards and never checks anything out (lib/workers/repository/process/index.ts:197-223).
So on a cache hit the working tree is left on the default branch (where syncGit put it), and the file:// read hits ENOENT. From there:
readLocalFile() → null → JSON.parse(null) → null → the JSONata transform gives undefined
→ ReleaseResultZod.parse throponse has failed validation", returns null) → getRawPkgReleases turns null into 'no-result' → lookupUpdates reports "Failed to look up … no-result"'t set caching, so the package cache doesn't hide it either.

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[RahulGautamSingh]

Sounds like a bug to me. I don't have a solution at the moment, so converting it to an issue.

[RahulGautamSingh]

#43890