Skip to content

Commit

Permalink
feat: Rely on local cache instead of npm global dependencies folder
Browse files Browse the repository at this point in the history
BREAKING CHANGE:
- `npm-cross-link` no longer interferes with npm global `node_modules` folder. For any reuse it relies on user local cache directory.
- No longer `npm link` is used internally. Any links are configured directly via symlinking
  • Loading branch information
medikoo committed Mar 24, 2024
1 parent dc2f43e commit d031c56
Show file tree
Hide file tree
Showing 17 changed files with 179 additions and 372 deletions.
34 changes: 13 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
[![Build status][build-image]][build-url]
[![npm version][npm-image]][npm-url]

_Note: Due to quirky way of how `npm link <package>` works in npm v7 (it manipulates also other not related dependencies in `node_modules`). this package at this point doesn't work well with npm v7. This issue will be addressed with next major release_

# npm-cross-link

## Automate `npm link` across maintained packages and projects
Expand All @@ -13,12 +11,14 @@ npm install -g npm-cross-link

### Use case

You maintain many npm packages and prefer to cross install them via `npm link`, handling such setup manually can be time taking and error-prone.
You maintain many distinct npm packages and prefer to cross link them across each other, handling such setup manually can be time taking and error-prone.

`npm-cross-link` is the npm installer that ensures all latest versions of dependencies are linked to global folder
and non latest are installed on spot (but with its deep dependencies located in its own `node_modules`)
`npm-cross-link` is packages installer which installs dependencies into `node_modules` the following way:

For maintained packages, it ensures its local installation is linked into global folder
- Dependencies which expect to rely on some peer dependencies, are placed directly in `node_modules`
- All other dependencies are linked:
- Maintained dependencies referenced by _latest_ versions are linked to its corresponding repository folders
- All others are installed once in dedicated cache folder, and are linked into that folder

### How it works?

Expand All @@ -29,28 +29,20 @@ When running `npm-cross-link -g <package-name>` for maintained package, or when
1. If package repository is not setup, it is cloned into corresponding folder (`~/npm-packages/<package-name>` by default). Otherwise optionally new changes from remote can be pulled (`--pull`) and committed changes pushed (`--push`)
2. All maintained project dependencies (also `devDependencies` and eventual `optionalDependencies`) are installed/updated according to same flow.

- Not maintained dependencies (not found in `packagesMeta`) if at latest version are ensured to be installed globally and npm linked to global npm folder. Otherwise they're installed on spot but with its dependencies contained in dependency folder (not top level node_modules).
- Maintained project dependencies (those found in `packagesMeta`) if referenced version matches local, are simply cross linked, otherwise they're istalled on spot (with its dependencies contained in dependency folder, not top level node_modules).

3. Package is ensured to be linked to global npm folder
- Not maintained dependencies (not found in `packagesMeta`) are ensured to be installed in cache and linked (unless they depend on some peer dependencies, then they're copied directly into `node_modules`)
- Maintained project dependencies (those found in `packagesMeta`) if referenced version matches local, are simply cross linked, otherwise they're linked from cache folder

All important events and findings are communicated via logs (level of output can be fine tuned via [LOG_LEVEL](https://github.com/medikoo/log/#log_level) env setting).

As each dependency is installed individually (and maintained packages are being installed recursively), the first install may appear slower from regular`npm` or `yarn` installs. Still it doesn't affect all further installs.

#### npm resolution

When relying on npm, it relies on version as accessible via command line.

If you rely on global Node.js installation, then Node.js update doesn't change location of global npm folder, so updates to Node.js are free from side effects when package links are concerned.

However when relying on [nvm](https://github.com/creationix/nvm), different npm is used with every different Node.js version, which means each Node.js/npm version points to other npm global folder. That's not harmful per se, but on reinstallation all links would be updated to reflect new path.

To avoid confusion it's better to rely on global installation. Still [nvm](https://github.com/creationix/nvm) is great for checking this project out (as then globally installed packages are not affected).
Internally `npm-cross-link` relies on `npm` being accesible to prepare cached versions of installed packages.

#### Limitations

All subdependencies of project dependencies are installed within dependencies `node_modules` folders. It means that if e.g. dependency `A` and dependency `B`, depend on same version of dependency `C`, (and they're not maintained packages, so they're either linked to global installation or installed on spot) they will use different installations of `C`.
All subdependencies of project dependencies are installed within dependencies `node_modules` folders. It means that if e.g. dependency `A` and dependency `B`, depend on same version of dependency `C`, (and they're not maintained packages, so they're either linked to cache or installed on spot) they will use different installations of `C`.

npm since early days ensured that in such scenarios `C` is installed top level (so it's shared among `A` and `B`), npm-cross-link doesn't ensure that.

Expand All @@ -68,7 +60,7 @@ Installs or updates given project dependencies. If dependency version is not spe

#### `npm-cross-link -g [...options] ...[<@scope>/]<name>`

Installs or updates given packages globally. Due to `npm-cross-link` installation rules it's only latest versions of packages that are globally linked.
Installs or updates given packages on its own. If it's maintained package, then it's ensured in resolved maintained folder, in all other cases packages is simply ensured to be installed in cache

#### `npm-cross-link-update-all [...options]`

Expand All @@ -78,7 +70,7 @@ Updates all are already installed maintained packages

- `--pull` - Pull eventual new updates from remote
- `--push` - For all updated packages push eventually committed changes to remote
- `--bump-deps` - (only non global installations) Bump version ranges of dependencies in `package.json`
- `--bump-deps` - (only non-global installations) Bump version ranges of dependencies in `package.json`
- `--no-save` - (only for dependencies install) Do not save dependency to `package.json` (effective only if its not there yet)
- `--dev` - (only for dependencies install) Force to store updated version in `devDependencies` section
- `--optional` - (only for dependencies install) Force to store updated version in `optionalDependencies` section
Expand Down Expand Up @@ -174,7 +166,7 @@ Installer by default removes all dependencies not referenced in package `package

#### `toBeCopiedDependencies`

Optional. Eventual list of non maintained dependencies that in all cases should be copied into `node_modules` and not linked to global installation.
Optional. Eventual list of non maintained dependencies that in all cases should be copied into `node_modules` and not linked to cache

[build-image]: https://github.com/medikoo/npm-cross-link/workflows/Integrate/badge.svg
[build-url]: https://github.com/medikoo/npm-cross-link/actions?query=workflow%3AIntegrate
Expand Down
27 changes: 9 additions & 18 deletions install-packages-globally.js
Original file line number Diff line number Diff line change
@@ -1,30 +1,21 @@
"use strict";

const ensureObject = require("es5-ext/object/valid-object")
, toPlainObject = require("es5-ext/object/normalize-options")
, ensurePackageName = require("./lib/ensure-package-name")
, ensureConfiguration = require("./lib/ensure-user-configuration")
, createProgressData = require("./lib/create-progress-data")
, installPackageGlobally = require("./lib/install-package-globally")
, installMaintainedPackage = require("./lib/install-maintained-package");
const toPlainObject = require("es5-ext/object/normalize-options")
, ensureConfiguration = require("./lib/ensure-user-configuration")
, createProgressData = require("./lib/create-progress-data")
, installPackageGlobally = require("./lib/install-package-globally")
, tokenizePackageSpecs = require("./lib/utils/tokenize-package-specs");

module.exports = (packageNames, userConfiguration, inputOptions = {}) => {
packageNames = Array.from(ensureObject(packageNames), ensurePackageName);
const packageSpecsData = tokenizePackageSpecs(packageNames);
userConfiguration = ensureConfiguration(userConfiguration);
inputOptions = toPlainObject(inputOptions);
const progressData = createProgressData();

const promise = packageNames.reduce(async (previousPromise, name) => {
const promise = packageSpecsData.reduce(async (previousPromise, packageContext) => {
await previousPromise;
progressData.topPackageName = name;
const isExternal = !userConfiguration.packagesMeta[name];
const packageContext = { name };
if (isExternal) {
return installPackageGlobally(
packageContext, userConfiguration, inputOptions, progressData
);
}
return installMaintainedPackage(
progressData.topPackageName = packageContext.name;
return installPackageGlobally(
packageContext, userConfiguration, inputOptions, progressData
);
}, Promise.resolve());
Expand Down
2 changes: 1 addition & 1 deletion lib/cache-package/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ module.exports = memoizee(
potentialMethod =>
(methodData = potentialMethod.isApplicable(name, version, externalContext))
);

const versionCacheName = await method.resolveCacheName(version, methodData);
log.info("%s cache name for %s is %s", name, version, versionCacheName);
const versionCachePath = versionCacheName && resolve(cachePath, name, versionCacheName);
Expand Down Expand Up @@ -51,6 +50,7 @@ module.exports = memoizee(
);
}
}
log.notice("prepared %s", `${ name }@${ version }`, versionCachePath);
if (!versionCachePath) return packageTmpDir;

await rename(packageTmpDir, versionCachePath, { intermediate: true });
Expand Down
2 changes: 1 addition & 1 deletion lib/cache-package/sem-ver.js
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ module.exports = {
delete pkgJson.devDependencies;
if (pkgJson.scripts) delete pkgJson.scripts.prepare;
await writeFile(pkgJsonPath, JSON.stringify(pkgJson));
await runProgram("npm", ["install", "--production"], {
await runProgram("npm", ["install", "--production", "--ignore-scripts"], {
cwd: tmpDir,
logger: log.levelRoot.get("npm:install")
});
Expand Down
6 changes: 0 additions & 6 deletions lib/install-maintained-package/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,11 @@ const optionalChaining = require("es5-ext/optional-chaining")
, rm = require("fs2/rm")
, NpmCrossLinkError = require("../npm-cross-link-error")
, getPackageJson = require("../get-package-json")
, getNpmModulesPath = require("../get-npm-modules-path")
, cleanupNpmInstall = require("../cleanup-npm-install")
, setupRepository = require("../setup-repository")
, resolveExternalContext = require("../resolve-external-context")
, removeNonDirectDependencies = require("../remove-non-direct-dependencies")
, resolveMaintainedPackagePath = require("../resolve-maintained-package-path")
, npmLink = require("./npm-link")
, finalize = require("./finalize");

module.exports = async (packageContext, userConfiguration, inputOptions, progressData) => {
Expand Down Expand Up @@ -56,7 +54,6 @@ module.exports = async (packageContext, userConfiguration, inputOptions, progres
packageContext.packageJson = getPackageJson(path);
}

packageContext.linkedPath = resolve(await getNpmModulesPath(), name);
await resolveExternalContext(packageContext, progressData);
if (
packageContext.externalContext.latestVersion &&
Expand All @@ -71,9 +68,6 @@ module.exports = async (packageContext, userConfiguration, inputOptions, progres
// Cleanup outcome of eventual previous npm crashes
await cleanupNpmInstall(packageContext);

// Link package
await npmLink(packageContext);

// Setup dependencies
// (cyclic module dependency, hence required on spot)
await require("../setup-dependencies")(
Expand Down
46 changes: 0 additions & 46 deletions lib/install-maintained-package/npm-link.js

This file was deleted.

87 changes: 39 additions & 48 deletions lib/install-package-globally.js
Original file line number Diff line number Diff line change
@@ -1,67 +1,58 @@
"use strict";

const { resolve } = require("path")
, log = require("log").get("npm-cross-link")
, wait = require("timers-ext/promise/sleep")
, rm = require("fs2/rm")
, NpmCrossLinkError = require("./npm-cross-link-error")
, getNpmModulesPath = require("./get-npm-modules-path")
, runProgram = require("./run-program")
, nonOverridableExternals = require("./non-overridable-externals")
, resolveExternalContext = require("./resolve-external-context");
const log = require("log").get("npm-cross-link")
, wait = require("timers-ext/promise/sleep")
, semver = require("semver")
, installMaintainedPackage = require("./install-maintained-package")
, NpmCrossLinkError = require("./npm-cross-link-error")
, resolveExternalContext = require("./resolve-external-context")
, resolveLocalContext = require("./resolve-local-context")
, cachePackage = require("./cache-package");

module.exports = async (packageContext, userConfiguration, inputOptions, progressData) => {
const { name } = packageContext;
const { name, versionRange } = packageContext;
const { packagesMeta } = userConfiguration;
if (packagesMeta[name]) {
throw new NpmCrossLinkError(
`Cannot install "${ name }" globally. It's not recognized as a maintained package`
);
}
if (nonOverridableExternals.has(name)) {
throw new NpmCrossLinkError(
`Cannot install "${ name }" globally. It should not be installed with npm-cross-link`
);
}
const isExternal = !packagesMeta[name];

packageContext.installationJobs = new Set();

// Ensure to emit "start" event in next event loop
await wait();
progressData.emit("start", packageContext);

const linkedPath = (packageContext.linkedPath = resolve(await getNpmModulesPath(), name));
if (!isExternal) {
const { ongoing, done } = progressData;
// Esure we have it installed locally
if (!ongoing.has(name) && !done.has(name)) {
await installMaintainedPackage({ name }, userConfiguration, inputOptions, progressData);
}

const externalContext = await resolveExternalContext(packageContext, progressData);
if (!externalContext) {
throw new NpmCrossLinkError(
`Cannot install "${ name }" globally. It's doesn't seem to be published`
if (!versionRange || versionRange === "latest") {
progressData.emit("end", packageContext);
return;
}

const { localVersion } = resolveLocalContext(
packageContext, userConfiguration, progressData
);
}
const { globallyInstalledVersion, latestVersion } = externalContext;
if (!latestVersion) {
throw new NpmCrossLinkError(
`Cannot install "${ name }" globally. There's no latest version tagged`

if (localVersion && semver.satisfies(localVersion, versionRange)) {
progressData.emit("end", packageContext);
return;
}
log.error(
"%s will have %s version installed externally as non latest version is referenced",
name, versionRange
);
}

// Lastest version supported, ensure it's linked
if (globallyInstalledVersion === latestVersion) return;
if (globallyInstalledVersion) {
packageContext.installationType = "update";
log.notice(
"%s outdated at global folder (got %s expected %s), upgrading", name,
globallyInstalledVersion, latestVersion
const externalContext = await resolveExternalContext(packageContext, progressData);
if (!externalContext) {
throw new NpmCrossLinkError(
`Cannot install "${ name }" globally. It's doesn't seem to be published`
);
} else {
packageContext.installationType = "install";
log.notice("%s not installed at global folder, linking", name);
}
// Global node_modules hosts outdated version, cleanup
await rm(linkedPath, { loose: true, recursive: true, force: true });

await runProgram("npm", ["install", "-g", `${ name }@${ latestVersion }`], {
logger: log.levelRoot.get("npm:link")
});

progressData.emit("end", packageContext);
await cachePackage(
name, packageContext.latestSupportedPublishedVersion || versionRange, externalContext
);
};
3 changes: 0 additions & 3 deletions lib/non-overridable-externals.js

This file was deleted.

16 changes: 3 additions & 13 deletions lib/resolve-external-context.js
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
"use strict";

const optionalChaining = require("es5-ext/optional-chaining")
, log = require("log").get("npm-cross-link")
, isDirectory = require("fs2/is-directory")
, semver = require("semver")
, getPackageJson = require("./get-package-json")
, getMetadata = require("./get-metadata");
const log = require("log").get("npm-cross-link")
, semver = require("semver")
, getMetadata = require("./get-metadata");

const getVersions = ({ externalContext: { metadata } }) => Object.keys(metadata.versions);

Expand All @@ -18,12 +15,6 @@ const resolveStableVersions = ({ externalContext: { metadata } }) =>
})
.map(([version]) => version);

const getGloballyInstalledVersion = async ({ linkedPath }) => {
// Accept installation only if in directory (not symlink)
if (!(await isDirectory(linkedPath))) return null;
return optionalChaining(getPackageJson(linkedPath), "version") || null;
};

const resolveLatestSupportedVersion = packageContext => {
const { dependentContext, name, versionRange, isSemVerVersionRange } = packageContext;
if (!isSemVerVersionRange) return null;
Expand All @@ -46,7 +37,6 @@ module.exports = async (packageContext, progressData) => {
const metadata = await getMetadata(name);
if (metadata) {
externals.set(name, {
globallyInstalledVersion: await getGloballyInstalledVersion(packageContext),
latestVersion: metadata["dist-tags"].latest,
latestHasPeers: Boolean(
metadata.versions[metadata["dist-tags"].latest].peerDependencies
Expand Down
Loading

0 comments on commit d031c56

Please sign in to comment.