URL parsing API

[robingustafsson]

Problem

Parsing URLs is a recurring need when working with HTTP and other protocols. There's currently no native API in k6 for so. As #991 (comment) mentions a jslib polyfill is available but a native API for a context like k6, it would be better.

Proposal

As the comment history suggests, we would like to implement all or most of the relevant parts for k6 of the URL interface as defined by the Web API.

import URL from "k6/url";

const myURL = new URL("https://user:pass@sub.example.com:8080/p/a/t/h?query=string#hash");

console.log(myURL.hostname);

Further reading

• https://developer.mozilla.org/en-US/docs/Web/API/URL
• https://url.spec.whatwg.org/#api
• https://nodejs.org/docs/latest/api/url.html
• https://caniuse.com/#feat=url

[na--]

I think we might get this when we update core-js: zloirock/core-js#454 @mstoykov, can you confirm?

Besides, I don't think these types of features would be best implemented in the k6 core Go (i.e.import URL from "k6/url"; ) code. Even if core-js doesn't provide this, it still seems better to do it as another JS polyfill.

[mstoykov]

I was thinking the same thing, but am currently debugging windows specific issue and don't want to checkout the code, but probably it will be done by core-js :)

[robingustafsson]

Ah nice, looks to be included in core.js yes 🎉

[ianwalter]

What's the status of this and can anyone suggest a solution/workaround in the meantime?

[na--]

@ianwalter, some URL polyfill for node.js might work in k6 without issues, if it doesn't depend on other node or browser APIs

[ianwalter]

Thats a great idea, thanks @na--

[nicosommi]

@na-- @ianwalter I'm also searching for this URL object... such a trivial library for load test and so hard to get it working.
Where you able to find a polyfill that doesn't use browser document object or node API?

[nicosommi]

ended up building my own minimal wrapper

[na--]

I am wondering whether I should close this, since it doesn't seem like a good idea to add this to the Go code of k6. Web APIs in general are not ECMAScript APIs, so I doubt we'll get a PR with them merged in goja.

I see that the URL and URLSearchParams polyfill from core.js was recently added to jslib.k6.io: https://github.com/loadimpact/jslib.k6.io/tree/master/lib/url/1.0.0

So is this enough? I guess we can also bundle it directly in k6, assuming there are no licensing issues, though that seems unnecessary when it's so easy to use already:

import { URL, URLSearchParams } from "https://jslib.k6.io/url/1.0.0/index.js"

const address = new URL('https://test.k6.io')
address.searchParams.append('hello', 'world')
console.log(address.toString()) // prints https://test.k6.io/?hello=world

const search = new URLSearchParams([
  ['foo', 'bar'],
  ['hello', 'world']
])
console.log(search.toString()) // prints foo=bar&hello=world

[robingustafsson]

I agree that the introduction of https://github.com/loadimpact/jslib.k6.io/tree/master/lib/url/1.0.0 solves most of this issue. The only reason why I'd still argue this should be included in k6 core is that URL parsing/composing feels like a very basic and core functionality of a load testing tool 🙂

Another argument is that we have users who need to run k6 in firewalled environments where no Internet-bound connections are allowed and thus using jslib.k6.io hosted libs becomes more of a involved process of having to download this kind of lib and use it as a local dependency. Not a major issue perhaps, but for sure inconveniant for such a basic (in the context of a load testing tool) thing as composing and parsing URLs.

[na--]

Fair enough, very good points! And taking a second look at the JS polyfill, 32kb of minified JavaScript for such a simple thing seems a bit excessive... 😱

So, unless that can be substantially reduced, we probably don't want to bundle it and have every VU parse these 32kb when it's initialized, now that we're so close to finally getting rid of core.js completely... So my vote is to write it in Go and try to upstream it in goja, but if that doesn't get accepted, we can keep it in k6, since you are right that as a load testing tool, it's not going to be out of place. In any case, the Go implementation should be relatively easy, we can base it on the net/url package - hopefully here aren't any big differences.

[na--]

FWIW, people are complaining about the weight of the URL polyfill from core.js (#1851), so another data point in favor of having a native Go version of it.

[mstoykov]

again ... I doubt the polyfill is heavy ... at least it wasn't for me ;)

[TimDaub]

I was disappointed when I learned that k6 is implemented in go.

E.g. that I get an error on URLSearchParams is confusing. In node, I'd expect compliance with e.g. whatwg specs. For k6 I don't know what to expect. How can I solve this problem?

[0000] ReferenceError: URLSearchParams is not defined                                             upted itat getParam (
file:///Users/timdaub/Projects/strikedao-stress-test/script.mjs:37:20(3))         iters  at register (file:///Users/timdaub
/Projects/strikedao-stress-test/script.mjs:59:22(40))

[na--]

@TimDaub, for now, by importing the polyfill linked above or some other URL JS polyfill.

As you can see from the comments in this issue, we want to have this specific API natively in k6 eventually, since it's so useful for our use case. However, I doubt we'll ever have full native compatibility with all WHATWG specs, since k6 is not a browser... node.js doesn't have that either! For example, it doesn't support the Fetch API, right? It has its own way of making HTTP requests that is different from browsers and from k6.

In the long run, we want to have full ECMAScript compatibility in k6 (though we have a lot of work remaining there, see #2296 (comment)), and we want to support the Web (WHATWG) APIs that make sense for our use case.

[Gikkman]

The recommended approach here yields a warning with the latest version of k6:

Save the following as script.js

import http from 'k6/http';
import { URL } from "https://jslib.k6.io/url/1.0.0/index.js"
import { sleep } from 'k6';

export const options = {
    vus: 1,
    duration: '10s',
};

export default function () {
    const appUrl = new URL('http://example.com');
    http.get(appUrl.toString())
    sleep(5);
}

Then run it through docker:

$ docker run --rm --net=host -i grafana/k6 run - <script.js  

time="2024-05-28T11:57:33Z" level=warning msg="The \"wrong\" path (\"file:///\") and the path actually used by k6 (\"file:///home/k6/\") to resolve \"https://jslib.k6.io/url/1.0.0/index.js\" are different. Please report to issue https://github.com/grafana/k6/issues/3534. This will not currently break your script but *WILL* in the future, please report this!!!"