v0.40.0 release notes

release notes/v0.40.0.md

@@ -0,0 +1,283 @@
+k6 v0.40.0 is here! This release includes:
+- Breaking changes to some undocumented and unintentional edge behaviors.
+- New experimental modules and first-class support for JavaScript classes.
+- Bugs and refactorings to pave the way for future features.
+
+Finally, the [Roadmap](#roadmap-and-future-plans) goes over the plans for the next cycles.
+
+## Breaking changes
+
+- [#2632](https://github.com/grafana/k6/pull/2632) During the refactoring to set tags to `metric.add` in the order they are provided, we discovered that you could provide tags as a key-value pair map *multiple* times in the same call. This was never the intended use and was never documented. As it was undocumented, and as such functionality makes no sense alongside every other API k6 has, we decided to remove this ability.
+- [#2582](https://github.com/grafana/k6/pull/2582) [For extensions using the event loop] Previously, if `RegisterCallback` result was called twice, the second call would silently break the event loop. This has never been expected behavior, and calling it twice is always a bug in the code using it. Now, calling the `RegisterCallback` result twice leads to a panic.
+- [#2596](https://github.com/grafana/k6/pull/2596) The `tainted` property of the Metric type is no longer outputted by the JSON output. That property was likely always going to have a `false` value as it was outputted at the beginning of the test.
+
+### Main module/script no longer pollutes the global scope [#2571](https://github.com/grafana/k6/pull/2571)
+
+During the ESM changes, we found that anything defined in the main module scope was also accessible globally. This was because it was directly evaluated in the global scope.
+This has now been remedied and is no longer the case. This is a *breaking* change, but given that the whole point of modules (CommonJS or ESM) is to separate them, this is obviously rather a bug than a feature.
+
+On that note, we've seen [reports](https://github.com/grafana/k6/issues/2623) by people who have this global accessibility of the main module (intentionally or not). Still, it seems relatively rare, with only a few usages in a script. So if you need to access something globally, our suggested workaround is to set it explicitly on the [global object `globalThis`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/globalThis).
+
+### `k6/ws` now respects the `throw` option [#2247](https://github.com/grafana/k6/issues/2247)
+
+`k6/http` has used the [`throw` option](https://k6.io/docs/using-k6/k6-options/reference/#throw) to figure out whether it should throw an exception on errors or return a response object with an error set on it (and log it).
+
+This functionality is finally also available for `k6/ws`, which previously would've *always* thrown an exception and thus involved more scripting in handling it ([#2616](https://github.com/grafana/k6/pull/2616)).
+
+This is a minor breaking change. By default, `throw` is `false`, so it now no longer throws an exception but instead returns a Response with `error` property.
+
+Thank you, @fatelei, for making this change!
+
+## New Features
+
+### Experimental modules [#2630](https://github.com/grafana/k6/pull/2630) and [#2656](https://github.com/grafana/k6/pull/2656)
+
+As mentioned in the v0.39.0 release notes, we're happy to announce that this release brings experimental modules. The main idea behind this initiative is to get community feedback earlier, which will help us improve them faster. We encourage you to try experimental modules out and provide feedback through the community forums or GitHub issues. 
+
+This release contains three experimental modules:
+- `k6/experimental/redis` - support for interaction with [Redis](https://redis.io/)
+- `k6/experimental/websockets` - a new Websockets API that copies the "web" [WebSocket API](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket)
+- `k6/experimental/timers` - adding [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout)/[`clearTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/clearTimeout) and [`setInterval`](https://developer.mozilla.org/en-US/docs/Web/API/setInterval)/[`clearInterval`](https://developer.mozilla.org/en-US/docs/Web/API/clearInterval) implementations.
+
+Important to highlight that the **k6 team does not guarantee backward compatibility for these modules** and may change or remove them altogether. Also, their import paths, starting with `k6/experimental`, will break when the modules stop being experimental. Of course, we are going to try to limit those breaking changes to a minimum and, when possible, do them in a backward compatible way for at least a version.
+
+<details>
+<summary> Redis example </summary>
+
+Here is a fairly big example using [xk6-redis](https://github.com/grafana/xk6-redis) as an experimental module to keep track of data in Redis:
+
+```javascript
+import { check } from "k6";
+import http from "k6/http";
+import redis from "k6/experimental/redis"; // this will be `k6/x/redis` if you are using it as extension
+import { textSummary } from "https://jslib.k6.io/k6-summary/0.0.1/index.js";
+
+export const options = {
+  scenarios: {
+    usingRedisData: {
+      executor: "shared-iterations",
+      vus: 10,
+      iterations: 200,
+      exec: "measureUsingRedisData",
+    },
+  },
+};
+
+// Instantiate a new redis client
+const redisClient = new redis.Client({
+  addrs: __ENV.REDIS_ADDRS.split(",") || new Array("localhost:6379"), // in the form of "host:port", separated by commas
+  password: __ENV.REDIS_PASSWORD || "",
+});
+
+// Prepare an array of crocodile ids for later use
+// in the context of the measureUsingRedisData function.
+const crocodileIDs = new Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9);
+
+export function setup() {
+  redisClient.sadd("crocodile_ids", ...crocodileIDs);
+}
+
+export function measureUsingRedisData() {
+  // Pick a random crocodile id from the dedicated redis set,
+  // we have filled in setup().
+  redisClient
+    .srandmember("crocodile_ids")
+    .then((randomID) => {
+      const url = `https://test-api.k6.io/public/crocodiles/${randomID}`;
+      const res = http.get(url);
+
+      check(res, {
+        "status is 200": (r) => r.status === 200,
+        "content-type is application/json": (r) =>
+          r.headers["Content-Type"] === "application/json",
+      });
+
+      return url;
+    })
+    .then((url) => redisClient.hincrby("k6_crocodile_fetched", url, 1));
+}
+
+export function teardown() {
+  redisClient.del("crocodile_ids");
+}
+
+export function handleSummary(data) {
+  redisClient
+    .hgetall("k6_crocodile_fetched")
+    .then((fetched) => Object.assign(data, { k6_crocodile_fetched: fetched }))
+    .then((data) =>
+      redisClient.set(`k6_report_${Date.now()}`, JSON.stringify(data))
+    )
+    .then(() => redisClient.del("k6_crocodile_fetched"));
+
+  return {
+    stdout: textSummary(data, { indent: "  ", enableColors: true }),
+  };
+}
+```
+
+This example also showcases how to write some data and clean up after yourself.
+</details>
+
+The extension does *not* run a Redis server. You need to separately handle running, scaling, and connecting infrastructure to Redis.
+
+The [xk6-redis repository](https://github.com/grafana/xk6-redis) has more examples, and the module is [documented in the official k6 documentation](https://k6.io/docs/javascript-api/k6-experimental-redis/).
+
+<details>
+<summary> WebSockets example </summary>
+
+This is a rewrite of the current WebSocket example at https://test-api.k6.io/.
+
+
+This showcases how a single VU can run multiple WebSockets connections asynchronously and how to stop them after a period using the timeout and interval functions.
+
+```javascript
+import { randomString, randomIntBetween } from "https://jslib.k6.io/k6-utils/1.1.0/index.js";
+import { WebSocket } from "k6/experimental/websockets"
+import { setTimeout, clearTimeout, setInterval, clearInterval } from "k6/experimental/timers"
+
+let chatRoomName = 'publicRoom'; // choose your chat room name
+let sessionDuration = randomIntBetween(5000, 60000); // user session between 5s and 1m
+
+
+export default function() {
+  for (let i = 0; i < 4; i++) {
+    startWSWorker(i)
+  }
+}
+
+function startWSWorker(id) {
+  let url = `wss://test-api.k6.io/ws/crocochat/${chatRoomName}/`;
+  let ws = new WebSocket(url);
+  ws.addEventListener("open", () => {
+    ws.send(JSON.stringify({ 'event': 'SET_NAME', 'new_name': `Croc ${__VU}:${id}` }));
+
+    ws.addEventListener("message", (e) => {
+      let msg = JSON.parse(e.data);
+      if (msg.event === 'CHAT_MSG') {
+        console.log(`VU ${__VU}:${id} received: ${msg.user} says: ${msg.message}`)
+      }
+      else if (msg.event === 'ERROR') {
+        console.error(`VU ${__VU}:${id} received:: ${msg.message}`)
+      }
+      else {
+        console.log(`VU ${__VU}:${id} received unhandled message: ${msg.message}`)
+      }
+    })
+
+
+    let intervalId = setInterval(() => {
+      ws.send(JSON.stringify({ 'event': 'SAY', 'message': `I'm saying ${randomString(5)}` }));
+    }, randomIntBetween(2000, 8000)); // say something every 2-8seconds
+
+
+    let timeout1id = setTimeout(function() {
+      clearInterval(intervalId)
+      console.log(`VU ${__VU}:${id}: ${sessionDuration}ms passed, leaving the chat`);
+      ws.send(JSON.stringify({ 'event': 'LEAVE' }));
+    }, sessionDuration);
+
+    let timeout2id = setTimeout(function() {
+      console.log(`Closing the socket forcefully 3s after graceful LEAVE`);
+      ws.close();
+    }, sessionDuration + 3000);
+
+    ws.addEventListener("close", () => {
+      clearTimeout(timeout1id);
+      clearTimeout(timeout2id);
+      console.log(`VU ${__VU}:${id}: disconnected`);
+    })
+  });
+}
+```
+</details>
+
+Note that no k6 iterations finish if any WebSocket is still open or if a timeout or an interval is not cleared or triggered. This means that your script must take care of clearing all intervals and closing the WebSocket at some point. However, k6 still kills the whole process if it takes too long to stop after the maximum test duration is reached.
+
+Current issues and future improvements for the WebSockets API can be found in its [issue tracker](https://github.com/grafana/xk6-websockets/issues). Currently, documentation is available through [MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket), though some features are not yet supported:
+- no Blob binary type - ArrayBuffer is the default
+- no `onMessage` and co. - only addEventListener
+
+### First-class support for JavaScript Classes
+
+As part of [updating goja](https://github.com/grafana/k6/pull/2610), k6 got native support for classes. Again, that's native, as in not by transpiling by the internal Babel.
+
+Because this actually implements classes as described in the latest ECMAScript specification, this also means we get a ton of additional class features that were never previously supported (for example, private fields). Additionally, at least one bug [#1763](https://github.com/grafana/k6/issues/1763) was fixed as a result of this, but probably many more as well.
+
+Due to this fairly significant change, some code could behave differently. Please report any issues, though consider that it's possible that the new behavior is just the correct one.
+
+Other updates from goja are:
+- optimizations around using strings and some access patterns
+- support for `\u{01234}` Unicode point syntax in regexp
+- Fixed a case where interrupting the VM did not work, especially around try/catch usage ([#2600](https://github.com/grafana/k6/pull/2600)). This was particularly problematic for k6, as it could lead to k6 hanging.
+
+Many thanks to @dop251 for continuing to improve goja! 
+
+#### New Test runtime for module extension developers [#2598](https://github.com/grafana/k6/issues/2598)
+
+While we develop extensions internally, we often need to repeatedly create the same structure. With the addition of the event loops, it is now required to set it up as well. Even k6 team members get parts of this wrong every once in a while, so we added a small type to be used by (extension) module developers to write tests easier ([#2602](https://github.com/grafana/k6/pull/2602)). 
+
+[This API](https://pkg.go.dev/go.k6.io/k6@v0.39.1-0.20220819115243-e09bb8727786/js/modulestest#Runtime) will likely change and evolve as we add more functionality or as we change the k6 internal API.
+
+## Bug fixes
+
+- [#2585](https://github.com/grafana/k6/pull/2585) `http.batch()` now displays an error if it is not given exactly 1 argument([#1289](https://github.com/grafana/k6/issues/1289)). Thanks, @vanshaj!
+- [#2596](https://github.com/grafana/k6/pull/2596) Fixes a potential data race in the JSON output. Includes a breaking change where `tainted` property is no longer outputted. That property was (likely) always going to have the value `false` as it was outputted at the beginning of the test.
+- [#2604](https://github.com/grafana/k6/pull/2604) Fixes SSL keylogger not working with absolute paths.
+- [#2637](https://github.com/grafana/k6/pull/2637) Fixes setting the options `rps` to `0` or below leading to exceptions. Now setting it to 0 or below disables the limit. Thanks, @tbourrely. [#2613](https://github.com/grafana/k6/issues/2613)
+- [#2278](https://github.com/grafana/k6/issues/2278) Reading `options.tags` directly was not possible. This was fixed by accident by [#2631](https://github.com/grafana/k6/pull/2631).  `k6/execution` is still the recommended way to access the [final options of the test](https://k6.io/docs/javascript-api/k6-execution/#get-test-options).
+
+## Maintenance and internal improvements
+
+- [#2590](https://github.com/grafana/k6/pull/2590) Updates direct dependencies without any interesting changes apart goja.
+- [#2591](https://github.com/grafana/k6/pull/2591) Changes to the CI process to always build rpm/deb and windows packages and use nfpm to do it. 
+- [#2593](https://github.com/grafana/k6/pull/2593) Internal cleanup after finally removing `common.Bind`.
+- [#2597](https://github.com/grafana/k6/pull/2597) Fixes go benchmarks we have broken over time.
+- [#2599](https://github.com/grafana/k6/pull/2599) Reformats `//nolint` comments as part of getting ready for go 1.19.
+- A bunch of fixes for tests [#2589](https://github.com/grafana/k6/pull/2589), [#2620](https://github.com/grafana/k6/pull/2620), [#2625](https://github.com/grafana/k6/pull/2625), [#2643](https://github.com/grafana/k6/pull/2543), [#2647](https://github.com/grafana/k6/pull/2647), [#2648](https://github.com/grafana/k6/pull/2648),
+- [#2607](https://github.com/grafana/k6/pull/2607) Fixes the build badge in the README. Thanks @AetherUnbound!
+- [#2614](https://github.com/grafana/k6/pull/2614) Fixes advice for RPM install on Amazon Linux.
+- [#2615](https://github.com/grafana/k6/pull/2615) Improves documentation of the RegisterCallback, following feedback on how hard it was to understand.
+- [#2627](https://github.com/grafana/k6/pull/2627) Create distinct test state objects for the pre-init and run phases.
+- [#2635](https://github.com/grafana/k6/pull/2635) Drop License header in each file.
+- [#2636](https://github.com/grafana/k6/pull/2636) Add [SECURITY.md](https://github.com/grafana/k6/security/policy) with instructions how to report security issues responsibly.
+- [#2641](https://github.com/grafana/k6/pull/2641) Fix spelling of `lose`. Thanks @spazm!
+- Update to golangci-lint v1.47.2 and enable a bunch more linters. [#2609](https://github.com/grafana/k6/pull/2609), [#2611](https://github.com/grafana/k6/pull/2611). Also, drop obsolete configurations [#2619](https://github.com/grafana/k6/pull/2619).
+
+## Roadmap and future plans
+
+This section discusses our plans for future versions. Notice that two big ticket items are here again―ESM modules and metric refactoring. They remain on the roadmap mostly for the sheer size of the work required on both, but also for some unforeseen needed changes, which actually should make them better in the long run. It also so happens that it is vacation season so the k6 team rightfully is taking some personal time away.
+
+### Native support for ECMAScript modules
+
+Native ESM support is coming. A [PR to k6](https://github.com/grafana/k6/pull/2563) is under development, and there's a branch that will become a PR to goja to add the support there. The k6 team is hopeful that this will land in the next version, v0.41.0!
+
+It turned out that there were a lot more things to be added as functionality - dynamic import, and the [tc39](https://tc39.es/) group released the latest ECMAScript specification adding support of top-level await in modules. While neither k6 nor goja has support for the `async`/`await` syntax, yet, this changes significantly the internal of the change, which did require a not insignificant refactor.
+
+Additionally, as previously mentioned, there were a bunch of changes to goja, including adding class support which also needed to be integrated.
+
+A future breaking change is that using CommonJS style exports along import/export syntax in the *same* file will no longer be possible.
+
+```javascript
+import http from "k6/http"
+
+exports.default = function() {} // this will start to error out
+```
+
+It will still be possible to import a CommonJS module and `require` an ES module or use `require` in an ES module. Having a dependency cycle mixing and matching CommonJS and ESM is also unlikely to work properly, but *might* do so in particular cases.
+
+This is really expected to have never been done by anyone as there isn't really much of a reason for this, but it is currently supported due to Babel transpiling everything to CommonJS behind the scenes.
+
+### Refactoring metrics
+
+The refactoring of metrics is underway, with a [PR](https://github.com/grafana/k6/pull/2594) for more performant internal representation. Unfortunately, this took longer to get to a stable state, and given the size and the type of change, the k6 team decided to *hold* merging it until very near the end of the cycle. It also doesn't have any noticeable change for most users. Instead, it will be merged early in the v0.41.0 release cycle, and then more changes to use it will be made through the release cycle.
+
+Some of those changes include supporting non-indexable tags, which also have a read [PR](https://github.com/grafana/k6/pull/2654). This change is also essential for historical reasons connected with how the `name`  tag works. As such, it also needed to be merged to release the internal metric refactor.
+
+Future breaking change: as part of the many changes and experiments, we found out that we can keep `url` as an indexable tag. Previously the plan was that it *along* `vu` and `iter` will become non-indexable. However, the `url` tag is *heavily* used and enabled by default. Because of that (and other internal reasons), the new change will be that `url` will stay indexable, but if used with `name` will be overwritten to have the same value as `name`. Arguably this is what most users would want in the case when they are using `name`. We plan to add a *non* indexable `raw_url` tag for those that do not. As such, we no longer will be [printing a warning when `url` is used](https://github.com/grafana/k6/pull/2655) in thresholds.
+
+Even so, we did make a bunch of changes to the internals of k6 that will pave the way forward ([#2629](https://github.com/grafana/k6/pull/2629), [#2631](https://github.com/grafana/k6/pull/2631)).
+
+We are also still working on incorporating the newly developed time series data model for the [Prometheus remote-write output extension](https://github.com/grafana/xk6-output-prometheus-remote). We are fixing bugs and improving the extension with the goal of eventually integrating it as a core built-in k6 output module in a future k6 release.
+
+[Distributed tracing support](https://github.com/grafana/k6/issues/2128) itself needs non-indexable tag support. Once that is merged, more work in that direction will be started.