inline SQL scripts for bundlers

[timelf123]

Description

Removing fs dependency and inlining SQL files into a TS file at build time before tsc is run allows this to be run in serverless environments. For example, this nextjs config in vercel worked:

  webpack: (config) => {
    config.module.rules.push(
      {
        test: [
          path.resolve("node_modules/pg/lib/native"),
          path.resolve("node_modules/import-fresh"),
          path.resolve("node_modules/graphile-worker/dist/getTasks"),
        ],
        loader: "null-loader",
      },
// unsure if this is needed after the inlining changes
      {
        test: path.resolve("node_modules/graphile-worker/dist/migrate"),
        loader: "@vercel/webpack-asset-relocator-loader",
        options: {
          outputAssetBase: "graphile-worker",
        },
      }
    );

    // Return the modified config object
    return config;
  },

Fixes #143

Performance impact

Front loaded at build time

Security impact

None

Checklist

• My code matches the project's code style and yarn lint:fix passes.
• I've added tests for the new feature, and yarn test passes. - current e2e tests work
• I have detailed the new feature in the relevant documentation.
• I have added this feature to 'Pending' in the RELEASE_NOTES.md file (if one exists).
• If this is a breaking change I've explained why.

[benjie]

Excellent start; here's some suggested changes 🙌

[benjie]

The escaping of the SQL wasn't safe in general, let's protect against future migrations. If you can think of any UTF-8 encoded string sql for which sql !== eval(escape(sql)) please let me know.

Suggested change

	const sqlModule = sqlFiles.reduce((acc, file) => {
	const sql = fs.readFileSync(path.join(sqlDir, file), "utf8");
	const varName = "sql_" + file.replace(".sql", "").replace(/-/g, "_");
	return acc + `export const ${varName} = \`${sql}\`;\n`;
	}, "");
	/**
	* Turns `str` into a String.raw expression, safely escaping backticks and
	* placeholder strings.
	*/
	function escape(str) {
	return (
	"String.raw`" +
	str
	.replace(/`/g, '` + "`" + String.raw`')
	.replace(/\$\{/g, "$$` + String.raw`{") +
	"`"
	);
	}
	const sqlModule =
	"/* This file is auto-generated by `npm run build:sql`; DO NOT EDIT */\n" +
	"// prettier-ignore\n" +
	"const migrations = {\n" +
	sqlFiles
	.map((file) => {
	const sql = fs.readFileSync(path.join(sqlDir, file), "utf8");
	return ` ${JSON.stringify(file)}: ${escape(sql)},`;
	})
	.join("\n") + "\n};\n";

[timelf123]

Unless I am misunderstanding, I believe this may break out, but isn't an SQL escape. I can do a bit of fuzzing
}; console.log('Hacked!'); //

[benjie]

Nice try, but that looks safe enough to me:

$ node
Welcome to Node.js v18.17.1.
Type ".help" for more information.
> function escape(str) {
...   return (
...     "String.raw`" +
...     str
...       .replace(/`/g, '` + "`" + String.raw`')
...       .replace(/\$\{/g, "$$` + String.raw`{") +
...     "`"
...   );
... }
undefined
> sql = "}; console.log('Hacked!'); //"
"}; console.log('Hacked!'); //"
> sql === eval(escape(sql))
true
> sql = "`}; console.log('Hacked!'); //`"
"`}; console.log('Hacked!'); //`"
> sql === eval(escape(sql))
true

[benjie]

When you get a chance, please regenerate the generated file (and remove the old version). Cheers!

[benjie]

This looks almost perfect. One last tiny change and regenerate and then we're good I think!

[timelf123]

This looks almost perfect. One last tiny change and regenerate and then we're good I think!

Thanks, changed!

[benjie]

Please regenerate the generated file 🙏

[benjie]

Looks great to me!

[benjie]

Released as graphile-worker@0.15.0 🙌

[timelf123]

@benjie thank you for the PRR and quick release/tag!