[Proposal]: npx replacement

[KilianKilmister]

UPDATE: npm-v7-beta just got released

In the blog post it's stated that npx as a package will be deprecated at release and it's functionality will be implemented directly into npm.
This is great news, as v7 looks awesome, but it also means this project just got a whole lot easier.

Background

npx is currently unmaintained (see: this issue), and it doesn't look like it will get support any time soon. Potential contributers are present, but there is no administration who could look at or merge PRs.

It'is a real shame, seeing at how useful of a tool it can be for both setting up and maintaining a project. It is also a really simple way of ensuring, that your tools are up to date.

npx also has some really neat, but not well known features (see: readme), including:

• -c <string>: Execute inside an npm run-script-like shell environment,
• --shell-auto-fallback [<shell>]: Generates shell code to override your shell's "command not found" handler with
  one that calls npx.
• --no-install: If passed to npx, it will only try to run <command> if it already exists in the current path or in
  $prefix/node_modules/.bin.
• --ignore-existing: If this flag is set, npx will not look in $PATH, or in the current package's node_modules/.bin for an
  existing version before deciding whether to install.
• -n, --node-arg <string>: Extra node argument to supply to node when binary is a node script.

Used elegantly in scripting npx is a powerful tool.

Summary

the aim is to provide an alternative for npx that is easy to use and has a very familiar feel. And to equip it with aditional functionality in and around project-managment. But the core-runner should be simple enough to be usable in production at times, for example to always use up-to-date dependancies.

Goals / Target-Specs

• include --shell-auto-fallback and make temporary usage easy (eg. repl or runFile)
• proper cleanup on failed install
• expose api for programmatic usage
  • include various utility methods methods for npm-related stuff
• expose a run-script-like shell
  • include the same env-vars as in package-scripts (include api-method to load them, too)
• ignore/update cached modules if a newer version is available (options to disable this)
• integrate many of the tools used by npm & co. to keep behaviour consistent and share cach etc.
• reserve possibility to integrate with package-services (see: below)

lower priority goals

• provide a shell for direct usage: some applications enter a shell-mode when no arguments are provided to the main
  command. in this mode you can access manuals and specify sub-commands directly (eg. an npm-shell would allow you use run <script>or install <package> directly without having to write npm <run/install> <script/package>
• include debug functionality for package-scripts
  • add basic benchmarking -tool
• integrate into package-services (see: below
• increase GitHub integration (see: below)
• include some Yarn functionalities (see: below)
  • workspaces
  • dependenciesMeta

Components

Subprojects

this can basically be devided into:

• a module that provides a run-script-like shell
• a module that checks for updated versions and things like that
• a utility module providing auxilliary functionalities
• the 2 main modules containing the cli and api

Similar Projects

• a yarn-specific-verison of npx, documentation is sparse and it's not used much, so i don't know how well it works

Tools

• npm itself
• the npm-project is also well split-up into smaller modules, many of which can be used for this project

Notes

Package Services

npm experimented with srevice-packages in the past, which instead of simlpy exposing a module or bin to use, provide micro-service-daemons that can equally specify dependencies and be used as dependency. the project was abandond years ago, but i think a similar principle could be implemented. I'm picturing something akin to ultra simple docker-container-like services that would be run not in a global engine-instance like docker, but on a per project basis node-instance (utilizing nodes cluster-module for multithreading), largely configurable in a package.json, communicating internally via IPC and can use ports/pipes/sockets to comunicate with thingd outside the project.

this is an entirely different project i would like to tackle at some point, but i would like to keep the option to integrate it at a later point

GitHub Integration

When GitHub aquired npm, they mentioned they were planning on migrating public packages to github and use npm for hosting private packages. this will generally happen in a seamless manner, but it has lead to github implementing some nice functionalities in their package system. it would be great to present them to users and make them as easy as possible to use. Could also possibly add github-actions functionality.

Yarn

While i do like some of the functionality of yarn, I don't think their plug-and-play aproach is the way to go for project dependencies. This project offers a different aproach with similar functionality, but unlike yarns PuP will be compatible with every npm-package by default.
It won't be as disk-I/O-efficient, but this could still be added using a package-service that uses active caching.

[KilianKilmister]

@karfau i got sick of the disorganised nature of my ideas about modernization, so i set up an Org, where these things can be centralized. this also makes it easier to work with different people on different projects. it's still very much work in progress, but this repo will serve as a good starting point for any new projects around these. Don't get scared off, it's for you to descide if and to what you want to contribute.

This thread contains a basic proposal about what i have planned for a potential npx alternative. They are quite project specific so i don't repeat going off the rails like last time.

Have a look at it and ask questions, propose ideas or raise issues. And if everything turns out well, we can look at starting a prototype.

[Den-dp]

You can describe your ideas as rfc in https://github.com/npm/rfcs
They are very open to good ideas and new people. 😉

[KilianKilmister]

@Den-dp i did check it out a while ago, and how they handle stuff is probaly one of the resons of npms success.

I can imagin some crossover happening in the future when esm and things like that become more common, but until that point, my ideas tend to be not the most popular ones among the more conservative programmers, as I'm radically advocating for modernising code-bases and dev environments (and as a consequence i'm not too concerened about backwards compatability and favour early deprecation and constant upgrading).

The plan is to integrate the package aspects of this project as natively with npm as feasable for my goals, and with that, it's not unlikely that i'll push for parts of this project (or ubgrades to the npm tools it uses) to be added to npm.
But as it stands, the plan is for this to serve as a fundament of modernizing dev-environments, so i need to be careful about things like dependency-hell.

I would love to contribute and collaborate, but my experience says that i'm too early with my ideas. Although since Node v14 dropped, the community has been rapidly progressing and warmed up to modernization, so it might not take too long.

Do you have any specific part in mind that might be suitable to implement into the npm ecosystem?
From the current RFCs there isn't too much up my ally, but this one does interest me, as i'm currently working on a sub-commandshell myself, but i do apreciate your suggestion.