JavaScript Webpack Vulnerability

[dilanbhalla]

Hello,

I had a question regarding the npm module webpack. Recently it was discovered that, since webpack contains a file that uses dangerous hash (md5.js), it may be considered unsafe. However, I would like to test for flow (dataflow and controlflow) in webpack to see if this md5 file is still even used/accessed, since if it is not then codeql could help determine if this is a false positive and safe to use.

Essentially where I am running into trouble is that when I download webpack (either from the open source repo or via npm) and create a codeql database of it, the node_modules directory (which contains the md5.js file) does not seem to be included within the database. In fact it seems like the database consists almost completely of files from codeql javascript libraries, webpack/lib, and webpack/test. At first I had realized that node_modules was included within .gitignore and I removed it from there (so now it shows up locally), however when I create a database from webpack to test for flow between md5.js and other parts of webpack, the node_modules directory still seems to be missing from the database. I tried to create a database of only the node_modules directory, but I got a build error saying that no JavaScript or TypeScript files were found, even though there are plenty of .js files in node_modules.

Would anyone be able to help me understand why codeql database create is not extracting files from node_modules, and what I might be able to do to fix this, so I can test for controlflow/dataflow from inside the node_modules directory to other parts of webpack?

Thanks!

[aibaars]

Be default CodeQL for JavaScript excludes node_modules folders. These folders normally contain the (transitive) dependencies of the source code to analyze and are often several orders of magnitude larger than the actual source code. It should be possible to include additional folders using the magic environment variables explained in

codeql/javascript/extractor/src/com/semmle/js/extractor/AutoBuild.java

Lines 75 to 207 in 3a9ffe1

	/**
	* An alternative entry point to the JavaScript extractor.
	*
	* <p>It assumes the following environment variables to be set:
	*
	* <ul>
	* <li><code>LGTM_SRC</code>: the source root;
	* <li><code>SEMMLE_DIST</code>: the distribution root.
	* </ul>
	*
	* <p>Additionally, the following environment variables may be set to customise extraction
	* (explained in more detail below):
	*
	* <ul>
	* <li><code>LGTM_INDEX_INCLUDE</code>: a newline-separated list of paths to include
	* <li><code>LGTM_INDEX_EXCLUDE</code>: a newline-separated list of paths to exclude
	* <li><code>LGTM_REPOSITORY_FOLDERS_CSV</code>: the path of a CSV file containing file
	* classifications
	* <li><code>LGTM_INDEX_FILTERS</code>: a newline-separated list of strings of form "include:PATTERN"
	* or "exclude:PATTERN" that can be used to refine the list of files to include and exclude.
	* <li><code>LGTM_INDEX_TYPESCRIPT</code>: whether to extract TypeScript
	* <li><code>LGTM_INDEX_FILETYPES</code>: a newline-separated list of ".extension:filetype" pairs
	* specifying which {@link FileType} to use for the given extension; the additional file type
	* <code>XML</code> is also supported
	* <li><code>LGTM_INDEX_XML_MODE</code>: whether to extract XML files
	* <li><code>LGTM_THREADS</code>: the maximum number of files to extract in parallel
	* </ul>
	*
	* <p>It extracts the following:
	*
	* <ol>
	* <li>all <code>*.js</code> files under <code>$SEMMLE_DIST/tools/data/externs</code> (cf. {@link
	* AutoBuild#extractExterns()};
	* <li>all source code files (cf. {@link AutoBuild#extractSource()}.
	* </ol>
	*
	* <p>In the second step, the set of files to extract is determined in two phases: the walking
	* phase, which computes a set of candidate files, and the filtering phase. A file is extracted if
	* it is a candidate, its type is supported (cf. {@link FileExtractor#supports(File)}), and it is
	* not filtered out in the filtering phase.
	*
	* <p>The walking phase is parameterised by a set of <i>include paths</i> and a set of <i>exclude
	* paths</i>. By default, the single include path is <code>LGTM_SRC</code>. If the environment
	* variable <code>LGTM_INDEX_INCLUDE</code> is set, it is interpreted as a newline-separated list of
	* include paths, which are slash-separated paths relative to <code>LGTM_SRC</code>. This list
	* <i>replaces</i> (rather than extends) the default include path.
	*
	* <p>Similarly, the set of exclude paths is determined by the environment variables <code>
	* LGTM_INDEX_EXCLUDE</code> and <code>LGTM_REPOSITORY_FOLDERS_CSV</code>. The former is interpreted
	* like <code>LGTM_INDEX_EXCLUDE</code>, that is, a newline-separated list of exclude paths relative
	* to <code>LGTM_SRC</code>. The latter is interpreted as the path of a CSV file, where each line in
	* the file consists of a classification tag and an absolute path; any path classified as "external"
	* or "metadata" becomes an exclude path. Note that there are no implicit exclude paths.
	*
	* <p>The walking phase starts at each include path in turn and recursively traverses folders and
	* files. Symlinks and most hidden folders are skipped, but not hidden files. If it encounters a
	* sub-folder whose path is excluded, traversal stops. If it encounters a file, that file becomes a
	* candidate, unless its path is excluded. If the path of a file is both an include path and an
	* exclude path, the inclusion takes precedence, and the file becomes a candidate after all.
	*
	* <p>If an include or exclude path cannot be resolved, a warning is printed and the path is
	* ignored.
	*
	* <p>Note that the overall effect of this procedure is that the precedence of include and exclude
	* paths is derived from their specificity: a more specific include/exclude takes precedence over a
	* less specific include/exclude. In case of a tie, the include takes precedence.
	*
	* <p>The filtering phase is parameterised by a list of include/exclude patterns in the style of
	* {@link ProjectLayout} specifications. There are some built-in include/exclude patterns discussed
	* below. Additionally, the environment variable <code>LGTM_INDEX_FILTERS</code> is interpreted as a
	* newline-separated list of patterns to append to that list (hence taking precedence over the
	* built-in patterns). Unlike for {@link ProjectLayout}, patterns in <code>LGTM_INDEX_FILTERS</code>
	* use the syntax <code>include: pattern</code> for inclusions and <code>exclude: pattern</code> for
	* exclusions.
	*
	* <p>The default inclusion patterns cause the following files to be included:
	*
	* <ul>
	* <li>All JavaScript files, that is, files with one of the extensions supported by {@link
	* FileType#JS} (currently ".js", ".jsx", ".mjs", ".cjs", ".es6", ".es").
	* <li>All HTML files, that is, files with with one of the extensions supported by {@link
	* FileType#HTML} (currently ".htm", ".html", ".xhtm", ".xhtml", ".vue", ".html.erb", ".jsp").
	* <li>All YAML files, that is, files with one of the extensions supported by {@link
	* FileType#YAML} (currently ".raml", ".yaml", ".yml").
	* <li>Files with base name "package.json" or "tsconfig.json", and files whose base name
	* is of the form "codeql-javascript-*.json".
	* <li>JavaScript, JSON or YAML files whose base name starts with ".eslintrc".
	* <li>All extension-less files.
	* </ul>
	*
	* <p>Additionally, if the environment variable <code>LGTM_INDEX_TYPESCRIPT</code> is set to "basic"
	* or "full" (default), files with one of the extensions supported by {@link FileType#TYPESCRIPT}
	* (currently ".ts" and ".tsx") are also included. In case of "full", type information from the
	* TypeScript compiler is extracted as well.
	*
	* <p>The environment variable <code>LGTM_INDEX_FILETYPES</code> may be set to a newline-separated
	* list of file type specifications of the form <code>.extension:filetype</code>, causing all files
	* whose name ends in <code>.extension</code> to also be included by default.
	*
	* <p>The default exclusion patterns cause the following files to be excluded:
	*
	* <ul>
	* <li>All JavaScript files whose name ends with <code>-min.js</code> or <code>.min.js</code>.
	* Such files typically contain minified code. Since LGTM by default does not show results in
	* minified files, it is not usually worth extracting them in the first place.
	* </ul>
	*
	* <p>JavaScript files are normally extracted with {@link SourceType#AUTO}, but an explicit source
	* type can be specified in the environment variable <code>LGTM_INDEX_SOURCE_TYPE</code>.
	*
	* <p>The file type as which a file is extracted can be customised via the <code>
	* LGTM_INDEX_FILETYPES</code> environment variable explained above.
	*
	* <p>If <code>LGTM_INDEX_XML_MODE</code> is set to <code>ALL</code>, then all files with extension
	* <code>.xml</code> under <code>LGTM_SRC</code> are extracted as XML (in addition to any files
	* whose file type is specified to be <code>XML</code> via <code>LGTM_INDEX_SOURCE_TYPE</code>).
	* Currently XML extraction does not respect inclusion and exclusion filters, but this is a bug, not
	* a feature, and hence will change eventually.
	*
	* <p>Note that all these customisations only apply to <code>LGTM_SRC</code>. Extraction of externs
	* is not customisable.
	*
	* <p>To customise the actual extraction (as opposed to determining which files to extract), the
	* following environment variables are available:
	*
	* <ul>
	* <li><code>LGTM_THREADS</code> determines how many threads are used for parallel extraction of
	* JavaScript files (TypeScript files cannot currently be extracted in parallel). If left
	* unspecified, the extractor uses a single thread.
	* <li><code>LGTM_TRAP_CACHE</code> and <code>LGTM_TRAP_CACHE_BOUND</code> can be used to specify
	* the location and size of a trap cache to be used during extraction.
	* </ul>
	*/

Another approach would be to rename the node_modules folder.