Explicitly defining NPM dependencies

[bemosior]

Per @pdfernhout:

...we made the choice to check in node_modules to avoid issue like the recent ones npm users have experienced with SSL certificates and also left-pad so we don't want to exclude them from git.

The above is a reasonable position to hold, and I wholeheartedly support it. However, is there value to additionally defining the dependencies at particular versions within package.json?

Doing so would cause the project's dependencies and their versions to be clearly listed in a single, coherent place. Besides making project dependency metadata clear, it would also enable an easier developer experience with dependency upgrades via npm. Both enable quick response to security issues in upstream dependencies. The primary benefit to me, however, is the clarity of project metadata.

The existence of multiple node_modules directories makes achieving the "single, coherent place" for project metadata more difficult. Since there is a single package.json, it might be worth an additional issue to discuss potential consolidation of checked-in dependencies and general project layout considerations necessary for such a change.

This may be a low-value item and therefore not worth pursuing. I raise this issue as part of the discussion in #15.

[pdfernhout]

@bemosior You have a good point about potential confusion by there being three node_modules directories (one at the top which was used to import libraries for the webapp, one for the Node.js server, and one for developer tools). It would be good to resolve this, but I'm not sure how yet. And when we do resolve this, I agree, it would be good to explicitly specify the dependencies in a package.json file. Below are some initial thoughts towards "potential consolidation of checked-in dependencies".

== More details

Essentially, there are three very different applications in the project:

• a web application frontend written in TypeScript
• a Node.js backend server in JavaScript
• a WordPress plugin backend written in PHP

Someday there may even be others, like a Drupal backend perhaps. Each has somewhat different needs. And the webapp frontend requires one or the other backend to function, so it is not stand-alone in that sense.

While the frontend webapp itself has an implicit dependency on some modules (including Mithril and D3 and various libraries which we copied to the webapp/lib directory), the only one checked in right now at the top level node_modules is RequireJS. I think you noted in issue #15 in a commit that RequireJS module version was inconsistent with the one in webapp/lib (although the commit is now gone with further updates there).

The Node.js server has the most dependencies, depending on several checked-in node modules. On whether to check in source for the modules for that, one other thing to consider is that an approach for managing npm dependencies for a library intended to be generally usable by many people in many situations may be different than that of an application which is expected to run in a specific setting.

Then beyond needs for runtime applications, are the needs for tools during development (ones we wrote). The tools have a few dependencies. And most, it not all, of those tools are not needed at this point anymore -- although they serve as templates for building new tools if needed.

Then there are tools during building. There is the TypeScript compiler as a current implicit dependency. There are maybe future ones for packaging or using alternative CSS syntax like with Stylus perhaps, or maybe Grunt or similar someday.

So, I agree with you that project layout could be improved. Right now, probably we could even get rid of the top level node_modules directory entirely (as it was only used to get a copy of RequireJS), and maybe the one for tools if we got rid of all the tools, and just leave the one for the node.js server. But, alternatively, we could maybe have the server refer to the top level node_modules directory which supported the server, the tools, and the webapp libraries needed for building the app. I have a feeling the latter is what we will end up with. Still, there is something about mixing dependencies for the webapp, the server, the developer tools, and the build process that bothers me -- but maybe that will grow on me over time.

Another issue down the road is improving the test infrastructure (perhaps with "Intern"). That will be another push to improve the project layout in some way. And likely there will be a node_modules directory related to testing tools. @cfkurtz might be working on an improved test infrastructure in the near future. That might be a good time to consolidate node_modules along lines you suggest.

Thanks again for bringing this issue up. We're going to continue to think about good approaches for this issue you raise. Feel free to suggest more alternatives or what you think would be best.

[bemosior]

Thank you for providing additional context. I understand what's going on now. My suggestion of consolidation would be inappropriate, because we truly have three separate but related projects (and potentially more in the future) together in one repository. We also have a separate issue with how we vendor dependencies (checking them in or otherwise hosting them internally), as it pertains to development versus production dependencies.

Multiple Projects

We have three distinct but related projects housed in one repository, which makes every decision related to operationalization many times more difficult. Otherwise one-dimensional problems - repository layout, dependency management, build scripts, and so on - now have two or more additional degrees of freedom because of the other related projects. What's best for one project can't always be what's best for another project and so on. I have witnessed similar circumstances get out of control and result in an indistinguishable "ball of mud" that feels impossible to maintain.

An elegant solution is to split the single repository into three and create dependent relationships between them, as appropriate. In this case, it makes sense to treat the common webapp frontend as a dependency of each of the two backends.

The Split

Let's say you create a GitHub organization called "NarraFirma" (which is an available name and free for open-source, by the way). You could then split the repository into its components, resulting in something along the lines of:

• NarraFirma/nf-typescript
• NarraFirma/nf-wordpress
• NarraFirma/nf-nodejs

(You can see examples of other organizations doing similar things here: 1, 2, 3)

To establish dependency relationships, we can make a decision within the context of each repository about how that takes place.

For the NarraFirma/nf-nodejs example, it makes sense to leverage existing npm constructs by asserting the dependency on NarraFirma/nf-typescript in package.json (see Git URLs as Dependencies). This wouldn't violate the vendoring rule, as you would be in control of the NarraFirma/nf-typescript git repository.

For the NarraFirma/nf-wordpress example, perhaps we shouldn't introduce npm at all. Perhaps it makes sense to mention the dependency in the README but provide it already baked into the binary download of a given wordpress plugin release. I don't know. The point is that you can make the best decision for that specific project.

Separating the projects also enables better fit for testing and development tool choices. With the right constraints and automation, the day-to-day development workflow doesn't have to change all that much. I'm happy to prototype this sort of change if you'd like to see what that world could look like. I'd also be happy to jump on a call and discuss it with you sometime, if that would be more helpful than the back-and-forth in these comments.

Vendoring: Dev versus Production Dependencies

Intermingling production and development dependencies can be "handled" through npm constructs, but that distinction is difficult if all (both dev and prod) dependencies are checked in. My off-the-cuff recommendation would be to subordinate the repository to production needs by checking in only production dependencies. Delegating control to npm for only development dependencies is a much less frightening prospect. You could also add binaries absent of dev-only dependencies within a release, but that might require too much per-release effort.

Miscellaneous

• The RequireJS issue was a problem I accidentally introduced and corrected in A few minor improvements. #15 (as seen in Changes from PR #15. #20).

[pdfernhout]

@bemosior Thanks for the additional suggestions. I'll have to think more over the next week about what your propose. Splitting the project into multiple repositories is certainly possible. But, it may to just create more confusion and development difficulties in practice through given the close coupling of the front-end and back-end -- creating more confusion than the issue of multiple node_modules file or even one shared node_module directory in terms of how to install or build the complete software. It might be a lot easier to just have one node_modules folder compared to three projects, with common versions for reach package for any that overlap. But I don't know for sure.

Rather than prototyping splitting up the project, it might be useful first to find and compare more examples of FLOSS projects (beyond the three you pointed to) that have a Node.js backend (or perhaps a backend in any language, even PHP or Java or Python) and also a complex JavaScript frontend to see how they do it. Especially of interest going forward is how projects set up their testing tools to test both frontend and backend (and also the combination). It would be helpful to know whether there is some consistent preferences for successful projects.

[bemosior]

Take your time! I'm happy if we leave this open for a bit and think on it.

A few links that reveal my bias (I admit to always separating front-end and back-end components when separate technologies and frameworks are involved):

• http://dontpanic.42.nl/2014/10/the-case-for-separating-front-and-back.html
• https://divshot.com/blog/opinion/every-web-app-is-two-apps/

Something a bit more balanced:
http://programmers.stackexchange.com/questions/107503/is-it-common-to-separate-back-end-and-front-end-into-two-positions-on-web-develo