Consolidate bundling

[bookmoons]

Combines external modules into a single compatibility layer bundle at convert time. Only bundles modules used in the output script. This should eliminate redundant code from shared dependencies of external modules.

Runs tinyify on the final bundle to optimize.

Closes #25.

[robingustafsson]

Looks good overall. I run into an issue though when I have collections with scripts that are using the shim'ed libs, eg. expect.js (https://github.com/bookmoons/postman-to-k6/blob/bookmoons/consolidate-bundling/lib/shim/expect.js). Then I get this error:
GoError: stat /<removed>/postman-to-k6/libs/chai.js: no such file or directory
The output script is piped to /<removed>/postman-to-k6/. Looks like the libs in https://github.com/bookmoons/postman-to-k6/blob/bookmoons/consolidate-bundling/lib/shim/ are trying to load unbundled files.

[bookmoons]

Shoot, will look into this.

[hiqqs]

This will remove the need for k6 scripts to import the library shims / not have those in the same repo correct?

[robingustafsson]

@johnhiggs No, this PR is for all shims and external libs to be put into one JS file rather than several different ones. It will also only include the external libs actually needed to run the output script, rather than all of them as is currently the case. Removing the need for external libs will require adding native support in k6 for these libs/APIs or by adding the relevant libs as curated libs to https://jslib.k6.io/ (can be used as remote modules or downloaded for use as local modules). The latter being the path we're currently aiming for.

[hiqqs]

Awesome thanks for clarifying

…

On Fri, Sep 20, 2019 at 12:02 PM Robin Gustafsson ***@***.***> wrote: @johnhiggs <https://github.com/johnhiggs> No, this PR is for all shims and external libs to be put into one JS file rather than several different ones. It will also only include the external libs actually needed to run the output script, rather than all of them as is currently the case. Removing the need for external libs will require adding native support in k6 for these libs/APIs or by adding the relevant libs as curated libs to https://jslib.k6.io/ (can be used as remote modules or downloaded for use as local modules). The latter being the path we're currently aiming for. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#41>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACTIN6IZYKPDR3Q46YM2V2LQKTXXTANCNFSM4IO56TIQ> .

[bookmoons]

Went over this more carefully. It has the problem you found and it was also not quite bundling everything. Pushed fixes for all of this.

But this has revealed an issue. One of the dependencies uses Object.setPrototypeOf(), which seems to be unavailable in k6. Going to look into this.

[robingustafsson]

Great, now it works better! 🎉

Regarding Object.setPrototypeOf(), maybe this would need an update of the core-js version we're using in k6. We've been trying to update core-js (and Babel.js) for some time now in k6, but I believe we ran into some issues with it. Let me check with the k6 team and I'll get back to you. If the update of core-js in k6 will take some time maybe we could polyfill ([1] or [2]) Object.setPrototypeOf() in postman-to-k6?

[1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/setPrototypeOf#Polyfill
[2] https://github.com/wesleytodd/setprototypeof

[bookmoons]

Nice, yes. Maybe that could also eliminate that rough edge in the shim, where it couldn't quite get a clean prototype.

[bookmoons]

Opened #43 for the prototype issue.

[bookmoons]

Made a submission that should resolve the prototype issue #44. After that's settled I can rebase this patch onto it.

[bookmoons]

Rebased this onto the prototype fix. Tests pass. The polyfill now ends up bundled in the compat layer. This collection that loads all shim components converts and executes successfully. I think it's in good shape.

{
	"info": {
		"_postman_id": "9b0cc8d3-04d1-4196-87a0-3af92541f760",
		"name": "Depends",
		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
	},
	"item": [
		{
			"name": "TestDepends",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "f3c1e91d-0c0c-481f-b841-a5d067d40a48",
						"exec": [
							""
						],
						"type": "text/javascript"
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "d7a05bac-89c9-4c48-aadc-472bc6631e6d",
						"exec": [
							"const cheerio = require('cheerio')",
							"const _ = require('lodash')",
							"const cryptoJs = require('crypto-js')",
							"",
							"const json = xml2Json(\"<html></html>\")",
							"pm.test('test body', function () {",
							"    pm.expect('Response body').to.include('Response')",
							"    pm.response.to.have.jsonSchema({})",
							"})",
							""
						],
						"type": "text/javascript"
					}
				}
			],
			"request": {
				"auth": {
					"type": "awsv4",
					"awsv4": [
						{
							"key": "secretKey",
							"value": "secret",
							"type": "string"
						},
						{
							"key": "accessKey",
							"value": "key",
							"type": "string"
						}
					]
				},
				"method": "GET",
				"header": [],
				"url": {
					"raw": "{{protocol}}://example.com?Action=List",
					"protocol": "{{protocol}}",
					"host": [
						"example",
						"com"
					],
					"query": [
						{
							"key": "Action",
							"value": "List"
						}
					]
				}
			},
			"response": []
		}
	],
	"variable": [
		{
			"id": "6148a217-db3c-4c2d-9807-992aab6ec48b",
			"key": "protocol",
			"value": "https",
			"type": "string"
		}
	],
	"protocolProfileBehavior": {}
}

[robingustafsson]

Yup, this works now. I have tested it as well 👍

I'd like to see one improvement though. Right now this is what the import list looks like in the output script from the Postman collection you posted above:

import "./libs/shim/core.js";
import "./libs/shim/cheerio.js";
import "./libs/shim/crypto-js.js";
import "./libs/shim/lodash.js";
import "./libs/shim/expect.js";
import "./libs/shim/jsonSchema.js";
import "./libs/shim/xml2Json.js";
import "./libs/shim/urijs.js";
import { aws4, URI } from "./libs/compat.js";

and all the shims (well, except libs/shim/core.js) follow this pattern:

import { URI } from '../compat.js'
const Extend = Symbol.for('extend')
postman[Extend].module.urijs = URI

could we instead inject these 3 line glue code snippets into libs/shim/core.js (an renaming the shim layer libs/shim.js)? So injecting something like this:

// inject as imports
import { cheerio, cryptoJS, lodash, expect, jsonSchema, xml2json, URI } from '../compat.js'

// this line would be redundant in libs/shim/core.js
const Extend = Symbol.for('extend')

// inject these at an appropriate place after Extend is defined
postman[Extend].module.cheerio = cheerio
postman[Extend].module.cryptoJS = cryptoJS
postman[Extend].module.lodash = lodash
postman[Extend].module.expect = expect
postman[Extend].module.jsonSchema = jsonSchema
postman[Extend].module.xml2json = xml2json
postman[Extend].module.urijs = URI

This would mean the libs output would only be 2 files, libs/shim.js and libs/compat.js, rather than all the small libs/shim/*.js files that are being output right now.

[bookmoons]

Will have a look at this.

[bookmoons]

As discussed offthread, I'll consolidate the shim to get all external deps into a single file.

[bookmoons]

I'm hitting issues here that are not yielding easily. There's some kind of interaction that seems to hit a memory leak. I'll have to put this aside for now.