Separate NxDaemon and NxCache to support use of shared cache - NX_CACHE_DIRECTORY

[jonhamm]

• I'd be willing to implement this feature (contributing guide)

Description

Could we not have NX_CACHE_DIRECTORY only hold cache artefacts and ~/node_modules/.cache/nxdeps/ (or NX_DEPS_DIRECTORY) hold nxdeps.json and then we could use the same directory as DAEMON_DIR_FOR_CURRENT_WORKSPACE ?

Motivation

When using NX_CACHE_DIRECTORY environment variable to share Nx cache directory among multiple git work-trees an error occurs when different work-trees are on different versions of the nxdeps.json file format - e.g.

> nx run ........

Invalid source or target versions. Source: 4.0, Target: 5.0.

Only backwards compatibility between "5.0" and "4.0" is supported.
This error can be caused by "@nrwl/..." packages getting out of sync or outdated project graph cache.
Check the versions running "nx report" and/or remove your "nxdeps.json" file (in ~/.nxcache/nx folder).

This will effectively make it impossible to have one common Nx Cache for all you work
NxCloud is not a solution as we do not want developers write access to the NxCloud

I believe it is wrong that nxdeps.json is stored in the cache directory - it should allways be stored in the workspace e.g. ~/node_modules/nxdeps/ ?
I also wondered if the current setup will work with NX_DAEMON as I see that

DAEMON_DIR_FOR_CURRENT_WORKSPACE = join(cacheDir, 'd');

and again this should be workspace local

Suggested Implementation

Alternate Implementations

[jonhamm]

This originated as a discussion - admittedly not attracting a lot of attention :-(
#8896

[jonhamm]

@vsavkin This was the issue ticket I referred to in our zoom meeting today

[minijus]

Error complaining about different versions of nxdeps.json is just a tip of the iceberg. nxdeps.json is a snapshot (collection of hashes) of current workspace.

When cache directory is shared among multiple git worktrees (i.e. in CI environment) what will happen is that there will be many parallel/colliding updates to nxdeps.json. Most likely nx affected/run-many will result in incorrect hashes, which in turn could result in false positive/negative run results.

We have exactly same limitation in our CI setup. This results in high saved hours on Nx Cloud as all cache hits must be downloaded from remote.

I have not tried yet, but there is a hypothesis that using emptyDir mount with subPath on nxdeps.json within cache directory mount could "isolate" nxdeps.json from whole shared cache directory.

containers:
- volumeMounts:
  - name: shared-cache
    mountPath: /tmp/shared-cache
  - name: nxdeps
    mountPath: /tmp/shared-cache/nxdeps.json
    subPath: nxdeps.json
volumes:
- name: shared-cache
  persistentVolumeClaim:
    claimName: shared-cache
- name: nxdeps
  emptyDir: {}

If this approach does not work, one can only hope for proper solution on Nx. That being said, I do like idea of having NX_DEPS_DIRECTORY environment variable.

Update: above mentioned "solution" with emptyDir does not work. It creates directory named nxdeps.json not a file. Also, trying to mount ConfigMap file does not work as such mount is not writable.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.