feat: collect unused files (#162)

* feat: unused files tracking

* benchmark: publish benchmark results from node_20.x

* benchmark: publish benchmark results from node_18.x

* feat: unused files tracking

* benchmark: publish benchmark results from node_20.x

* benchmark: publish benchmark results from node_18.x

* feat: show unused dependencies in the cli

* fix: config propagation

* docs: add changeset

* feat: add unused http api

* docs: unused

* benchmark: publish benchmark results from node_20.x

* benchmark: publish benchmark results from node_18.x

---------

Co-authored-by: skott_bot <skott.devtool@gmail.com>

.changeset/silver-spies-begin.md

@@ -0,0 +1,14 @@
+---
+"skott": minor
+---
+
+Allow unused files to be tracked and reported. From the CLI, `--showUnusedFiles` can be used to report unused files. From the API, a new `collectUnusedFiles` method is accessible through the graph API:
+
+```js
+import skott from "skott";
+
+const instance = await skott();
+const unusedFiles = instance.useGraph().collectUnusedFiles();
+```
+
+This version also includes a fix for a bug related to `--trackBuiltinDependencies` and `--trackThirdPartyDependencies` that were not propagated anymore (since 0.34.0) when being provided from the CLI.

README.md

@@ -21,7 +21,11 @@
 
 ✅ Deeply detects **circular dependencies** in an efficient way, with the ability to provide a max depth for the search
 
-✅ Many **builtin visualization modes** including a web application or terminal-based outputs such as file-tree or graph views. Visualization modes can be rendered using both the CLI and API.
+✅ Detect **unused source code files**. Eliminate dead code by finding files not imported anywhere else in the graph.
+
+✅ Detect **unused npm third-party dependencies**. Note that all unused `devDependencies` are not guaranteed to be detected as `depcheck` [only provides analysis for set of supported libraries](https://github.com/depcheck/depcheck) (eslint, karma, mocha, etc).
+
+✅ Many **builtin visualization modes** including a web application or terminal-based outputs such as file-tree or graph views. Visualization modes can be rendered using both the CLI and programatically using the API.
 
 ✅ Builtin **watch mode** updating the graph when file changes are detected. It works with all display modes (webapp and all CLIs visualization modes). Support all options of file ignoring/filtering from skott.
 
@@ -31,11 +35,9 @@
 
 ✅ Works with any custom **dependency resolver** (useful for specific monorepos integration where module identifiers need to be mapped to a specific workspace package) 
 
-✅ Detect **unused npm third-party dependencies**. Note that all unused `devDependencies` are not guaranteed to be detected as `depcheck` [only provides analysis for set of supported libraries](https://github.com/depcheck/depcheck) (eslint, karma, mocha, etc).
-
 ✅ Deeply **collect all dependencies of the project graph**, including third-party and builtin.
 
-✅ Deep **parent and child dependencies traversals** using DFS and BFS algorithms.
+✅ Graph API including deep **parent and child dependencies traversals** using DFS and BFS algorithms.
 
 ✅ Metadata collection per traversed node (file size, dependencies)
 
@@ -158,8 +160,9 @@ _Dead code_ can be defined as a code literally having no impact on the applicati
 
 However, tree shaking is not an easy task and can mostly work with module systems using static-based imports/exports such as ECMAScript modules. To avoid removing code that appears to be used at runtime, module bundlers are being very precise about determining automatically chunks of code that can be safely removed. Module bundlers can also be helped by providing them manually clues about what can be safely removed e.g. `/*#__PURE__*/` for Webpack.
 
-If you're not using tools implementing tree shaking, you will be able soon to use **skott**, which will bring up soon unused imports/exports warnings 🚀  
+Also, bundling might not be possible or might not even be a target. In that context, it's even more important to care about dead code elimination. Dead code can harm cold start and have unwanted side-effects.
 
+**skott** exposes information that can help identifying dead code and getting rid of it. Check documentation to get more information about identifying unused files and dependencies. 
 
 ## Graph Management
 

packages/skott/README.md

@@ -157,6 +157,14 @@ Using this command, skott will deeply search for all ".ts" and ".tsx" files star
 $ skott --fileExtensions=.ts,.tsx
 ```
 
+**Finding unused files and dependencies:**
+
+```bash
+$ skott --showUnusedFiles --showUnusedDependencies --trackThirdPartyDependencies
+```
+
+An important description of that feature is available below in the API section.
+
 **skott** offers many ways to visualize the generated graph.
 
 **Embedded Web Application**
@@ -246,6 +254,7 @@ const {
   traverseFiles, 
   collectFilesDependencies, 
   collectFilesDependingOn, 
+  collectUnusedFiles,
   findLeaves, 
   findCircularDependencies, 
   hasCircularDependencies 
@@ -258,6 +267,8 @@ const {
 const { useGraph } = await skott();
 const { traverseFiles } = useGraph();
 
+const unusedFiles = collectUnusedFiles();
+
 // Starting from any node, walking the whole graph
 for(const file of traverseFiles()) {
   // SkottNode { }
@@ -353,6 +364,23 @@ console.log(collectFilesDependencies("parent.js", CollectLevel.Shallow));
 // logs [ SkottNode { id: "children.js" } ]
 ```
 
+### Find unused files
+
+skott provides a way to collect **unused** files. Files are marked as "unused" from a pure source code analysis standpoint, meaning that a given file is considered unused only if it is not importing any other file and there is no other file importing it. In the graph lingo, we refer to these nodes as **isolated nodes**.
+
+Note: having a file being marked as unused does not necessarily mean that this file is useless, but rather than skott didn't find any use of it when traversing the whole project graph. Sometimes files are being exported as a npm library entrypoint even though they are not used in the internals of that library (for instance `package.json#exports`), or sometimes files are being used by other tools being run from npm scripts or whatever else toolchain.
+
+Unlike `unused dependencies` shown below, `unused files` don't need further analysis or need additional context e.g. a manifest file (package.json for Node.js) to be determined. This is why they belong in the Graph API, as `unused files` are nothing but `isolated nodes` in the context of skott.
+
+```javascript
+import skott from "skott";
+
+const { useGraph } = await skott();
+
+const unusedFiles = useGraph().collectUnusedFiles();
+// [ "index.js", "some-other-file.ts", "else.js" ]
+```
+
 ### Find unused dependencies
 
 skott provides a way to walk through dependencies listed in the current working directory manifest (package.json) and compare them to what it founds and marked as "used" during the analysis. The "use" marking will be done when a third-party module appears to be imported in the source code that was walked. All the third-party dependencies that are not used in the traversed files will be returned as "unused".

packages/skott/bin/cli.ts

@@ -154,6 +154,11 @@ cli
     "Search for unused third-party dependencies in the graph",
     false
   )
+  .option(
+    "-uf, --showUnusedFiles",
+    "Search for unused files in the graph",
+    false
+  )
   .option("-vb, --verbose", "Enable verbose mode. Display all the logs", false)
   .option(
     "-w, --cwd <path>",

packages/skott/src/graph/api.ts

@@ -0,0 +1,144 @@
+import type { DiGraph } from "digraph-js";
+
+import type { SkottConfig } from "../skott.js";
+
+import type { SkottNode } from "./node.js";
+
+export const CollectLevel = {
+  Deep: "deep",
+  Shallow: "shallow"
+} as const;
+
+export type CollectLevelValues =
+  (typeof CollectLevel)[keyof typeof CollectLevel];
+
+const skottToDiGraphTraversal = {
+  deepFirst: "dfs",
+  shallowFirst: "bfs"
+} as const;
+
+export class GraphApi<T> {
+  constructor(
+    private readonly graph: DiGraph<SkottNode<T>>,
+    private readonly config: SkottConfig<T>
+  ) {
+    this.getFileNode = this.getFileNode.bind(this);
+    this.getNodes = this.getNodes.bind(this);
+    this.traverseFiles = this.traverseFiles.bind(this);
+    this.collectFilesDependencies = this.collectFilesDependencies.bind(this);
+    this.collectFilesDependingOn = this.collectFilesDependingOn.bind(this);
+    this.collectUnusedFiles = this.collectUnusedFiles.bind(this);
+    this.hasCircularDependencies = this.hasCircularDependencies.bind(this);
+    this.findCircularDependencies = this.findCircularDependencies.bind(this);
+    this.findLeaves = this.findLeaves.bind(this);
+  }
+
+  getFileNode(id: string): SkottNode<T> {
+    return this.getNodes()[id];
+  }
+
+  getNodes() {
+    return this.graph.toDict();
+  }
+
+  *traverseFiles(options?: {
+    rootFile?: string;
+    moduleImportsCollection?: "deepFirst" | "shallowFirst";
+  }): Generator<SkottNode<T>, void, void> {
+    const rootNode = options?.rootFile;
+    const moduleImportsCollection =
+      options?.moduleImportsCollection ?? "shallowFirst";
+
+    const traversal = skottToDiGraphTraversal[moduleImportsCollection];
+
+    if (rootNode) {
+      return yield* this.graph.traverse({
+        rootVertexId: rootNode,
+        traversal
+      });
+    }
+
+    return yield* this.graph.traverse({
+      traversal
+    });
+  }
+
+  collectFilesDependencies(
+    rootFile: string,
+    collectLevel: CollectLevelValues
+  ): SkottNode<T>[] {
+    if (collectLevel === CollectLevel.Shallow) {
+      return this.graph.getChildren(rootFile);
+    }
+
+    const nodes = this.getNodes();
+    const childrenIds = Array.from(this.graph.getDeepChildren(rootFile));
+    const dependencies = [];
+
+    for (const id of childrenIds) {
+      dependencies.push(nodes[id]);
+    }
+
+    return dependencies;
+  }
+
+  collectFilesDependingOn(
+    rootFile: string,
+    collectLevel: CollectLevelValues
+  ): SkottNode<T>[] {
+    if (collectLevel === CollectLevel.Shallow) {
+      return this.graph.getParents(rootFile);
+    }
+
+    const nodes = this.getNodes();
+    const parentIds = Array.from(this.graph.getDeepParents(rootFile));
+    const dependingOn = [];
+
+    for (const id of parentIds) {
+      dependingOn.push(nodes[id]);
+    }
+
+    return dependingOn;
+  }
+
+  collectUnusedFiles(): Array<SkottNode["id"]> {
+    const leaves = this.findLeaves();
+    const unused = [];
+
+    for (const leaf of leaves) {
+      const node = this.getFileNode(leaf);
+      const noNodesDependingOn =
+        this.collectFilesDependingOn(leaf, CollectLevel.Deep).length === 0;
+      if (noNodesDependingOn) {
+        unused.push(node.id);
+      }
+    }
+
+    return unused;
+  }
+
+  hasCircularDependencies(): boolean {
+    return this.graph.hasCycles({
+      maxDepth: this.config.circularMaxDepth ?? Number.POSITIVE_INFINITY
+    });
+  }
+
+  findCircularDependencies(): Array<Array<SkottNode["id"]>> {
+    return this.graph.findCycles({
+      maxDepth: this.config.circularMaxDepth ?? Number.POSITIVE_INFINITY
+    });
+  }
+
+  findLeaves(): Array<SkottNode["id"]> {
+    const nodes = this.getNodes();
+    const leaves = [];
+
+    for (const node of Object.values(nodes)) {
+      if (node.adjacentTo.length === 0) {
+        leaves.push(node.id);
+      }
+    }
+
+    return leaves;
+  }
+}

packages/skott/src/rendering/config.ts

@@ -0,0 +1,15 @@
+import kleur from "kleur";
+
+import type { InputConfig, RuntimeConfig } from "../config.js";
+import { createRuntimeConfig } from "../instance.js";
+
+export function toRuntimeConfigOrDie<T>(input: InputConfig<T>): RuntimeConfig {
+  try {
+    return createRuntimeConfig(input);
+  } catch (error) {
+    // @ts-expect-error
+    console.log(`\n ${kleur.bold().red(error.message)}`);
+
+    return process.exit(1);
+  }
+}