POC: Add telemetry #4452

yannickcr · 2020-07-21T15:45:24Z

Summary

This PR is a POC for the telemetry project.

You can read the related RFC here: https://github.com/algolia/instantsearch-rfcs/blob/telemetry/accepted/telemetry.md

And see the telemetry payload in the request done by the e-commerce demo: https://deploy-preview-4452--instantsearchjs.netlify.app/examples/e-commerce/

codesandbox-ci · 2020-07-21T15:48:43Z

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 522df84:

Sandbox	Source
InstantSearch.js	Configuration

src/widgets/configure-related-items/configure-related-items.ts

Co-authored-by: Haroen Viaene <hello@haroen.me>

Haroenv

some very interesting ideas!

Haroenv · 2020-07-21T16:21:20Z

src/widgets/hits/hits.tsx

-  return makeHits({ escapeHTML, transformItems });
+  return {
+    ...makeHits({ escapeHTML, transformItems }),
+    $$params: widgetOptions,


instead we should forward $$params to the widget itself from the makeWidget call:

return makeHits({ escapeHTML, transformItems, $$params: widgetOptions });

Yes that's also a possibility, I will update the implementation to do this.

src/widgets/index/index.ts

Haroenv · 2020-07-21T16:22:27Z

src/widgets/index/index.ts

+    val = +val || 0;
+
+    if (val > 0xfffffff || val < 0) throw new Error('Unsupported');
+
+    // realloc
+    let length = this.length || 16;
+    while (length < this.pos + 4) length *= 2;
+    if (length !== this.length) {
+      const buf = new Uint8Array(length);
+      buf.set(this.buf);
+      this.buf = buf;
+      this.length = length;
+    }
+
+    this.buf[this.pos++] = (val & 0x7f) | (val > 0x7f ? 0x80 : 0);
+    if (val <= 0x7f) return;
+    this.buf[this.pos++] = ((val >>>= 7) & 0x7f) | (val > 0x7f ? 0x80 : 0);
+    if (val <= 0x7f) return;
+    this.buf[this.pos++] = ((val >>>= 7) & 0x7f) | (val > 0x7f ? 0x80 : 0);
+    if (val <= 0x7f) return;
+    this.buf[this.pos++] = (val >>> 7) & 0x7f;


what does this code mean? It's quite dense

Oh yeah, do not focus too much on it. It is a stripped down version of pbf (a protobuf implementation) with only the methods that are useful to us to keep it as light as possible (since the original implementation is not tree-shakable).

src/widgets/index/index.ts

Haroenv · 2020-07-21T16:26:33Z

src/widgets/index/index.ts

@@ -25,6 +25,82 @@ import {
  mergeSearchParameters,
 } from '../../lib/utils';

+import { WidgetType, WidgetParams, Schema } from '../../../schema';
+
+function Pbf() {


I like that we didn't need to use a dependency here, but I'd probably prefer it in its own file, and with tests

Agree, but for now in this POC I just made it "works" 😉, then is still tons of improvement in quality/structure/testing to do.

schema.proto

Haroenv · 2020-07-21T16:30:11Z

URLs have a limit of 500± characters in some cases, and we will still want to avoid going over that, however the ecom header is:

https://latency-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(4.1.0)%3B%20Browser%20(lite)%3B%20instantsearch.js%20(4.7.0)%3B%20JS%20Helper%20(3.1.1)&x-algolia-api-key=6be0576ff61c053d5f9a3225e2a90f76&x-algolia-application-id=latency&x-algolia-telemetry=qh8IwCUMyiUC6AeqHxLAJRrKJQz%2FB4cIiAiJCPkH8QeqHwrAJQnKJQT9B%2FEHqh8NwCUEyiUE8Qf5B9AlAaofDMAlBcolBooIiwiMCKofDMAlHsolBv8H%2BQfxB6ofCsAlC8olBPEHgwiqHxLAJRDKJQyNCIQIjgiPCPkH8QeqHw7AJRjKJQj%2FB5AIkQjxB6ofCsAlCsolBPEH%2BQeqHwzAJRnKJQb%2FB%2FkH8QeqHwrAJR3KJQTxB%2FkHqh8MwCUbyiUG8QfyB%2FkHqh8MwCUcyiUG8Qf%2FB4MI

(which is ±600 characters)

Is there some way this can be smaller?

I also notice that there's URL encoded parts in there (%2F, a.k.a /), is that expected?

Haroenv · 2020-07-21T16:32:53Z

src/widgets/index/index.ts

+
+        if (telemetryHeader !== newTelemetryHeader) {
+          telemetryHeader = newTelemetryHeader;
+          instantSearchInstance.client.transporter.queryParameters[


this is only v4 and our client compatible, we will want to send that as a query parameter

This part is very hacky here:

It should be in headers, not in queryParameters

instantSearchInstance.client.transporter.queryParameters is typed at read-only, so to do it properly we'll need to have a proper method to update it

I was not planning to be compatible with the client v3 (since we'll need to do some update to the client to properly set the header). Do you thing this is something we should do?

unknown request parameters of the client will go to either headers or query parameters (v4):

search([], { someParameter: "value", headers: { someHeader: "value })

v3 doesn't have any options for that, but at least it should ignore the second parameter

Haroenv · 2020-07-21T16:34:11Z

What is the process of decoding this string? I know you should be able to do decodeURIComponent(header).split('/'), but I don't know what to learn out of those strings :)

Co-authored-by: Haroen Viaene <hello@haroen.me>

yannickcr · 2020-07-22T12:37:12Z

URLs have a limit of 500± characters in some cases, and we will still want to avoid going over that, however the ecom header is:

https://latency-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(4.1.0)%3B%20Browser%20(lite)%3B%20instantsearch.js%20(4.7.0)%3B%20JS%20Helper%20(3.1.1)&x-algolia-api-key=6be0576ff61c053d5f9a3225e2a90f76&x-algolia-application-id=latency&x-algolia-telemetry=qh8IwCUMyiUC6AeqHxLAJRrKJQz%2FB4cIiAiJCPkH8QeqHwrAJQnKJQT9B%2FEHqh8NwCUEyiUE8Qf5B9AlAaofDMAlBcolBooIiwiMCKofDMAlHsolBv8H%2BQfxB6ofCsAlC8olBPEHgwiqHxLAJRDKJQyNCIQIjgiPCPkH8QeqHw7AJRjKJQj%2FB5AIkQjxB6ofCsAlCsolBPEH%2BQeqHwzAJRnKJQb%2FB%2FkH8QeqHwrAJR3KJQTxB%2FkHqh8MwCUbyiUG8QfyB%2FkHqh8MwCUcyiUG8Qf%2FB4MI

(which is ±600 characters)

For what I know in browsers URLs max length is 2048 characters, is there a limitation in the API I'm not aware about ?

The plan is to put it in a HTTP header (which have a limit around 8K), but in this POC I put it in the query parameters since I had some issues sending an unknown header to the API, I need to dig in it. Sending a custom HTTP header forces us to send a preflight request before the actual search request, so this is not a viable solution.

Also I planned to have a limit so we do not send a payload if it is too big.

Is there some way this can be smaller?

I used protobuf to specifically get the smallest possible payload. To get an even smaller one I see 2 solutions:

Find a better format than protobuf
Send less information

I also notice that there's URL encoded parts in there (%2F, a.k.a /), is that expected?

Yes, we're sending a base64 encoded string and it can contains some /, it should not be an issue.

What is the process of decoding this string? I know you should be able to do decodeURIComponent(header).split('/'), but I don't know what to learn out of those strings :)

To decode the string you'll need 2 things:

A protobuf implementation (there are some in multiple language)
The schema (schema.proto) to know how to decode it.

Here's how to do it in JS:

import Pbf from 'pbf';
import { WidgetType, WidgetParams, Schema } from '../schema';

const telemetryHeader =
  'qh8IwCUMyiUC6AeqHxLAJRrKJQzxB%2F8HhwiICIkI%2BQeqHwrAJQnKJQTxB%2F0Hqh8NwCUEyiUE8Qf5B9AlAaofDMAlBcolBooIiwiMCKofDMAlHsolBvEH%2Fwf5B6ofCsAlC8olBPEHgwiqHxLAJRDKJQzxB40IhAiOCI8I%2BQeqHw7AJRjKJQjxB%2F8HkAiRCKofCsAlCsolBPEH%2BQeqHwzAJRnKJQbxB%2F8H%2BQeqHwrAJR3KJQTxB%2FkHqh8MwCUbyiUG8QfyB%2FkHqh8MwCUcyiUG8Qf%2FB4MI';

// Decode the header and convert it to an Uint8Array
const arrayBuffer = new Uint8Array(
  Buffer.from(decodeURIComponent(telemetryHeader), 'base64')
);

// Parse the buffer using the schema
const pbf = new Pbf(arrayBuffer);
const output = Schema.read(pbf);

// Map the IDs to the matching strings
const mappedOutput = output.widgets.map(widget => ({
  type: Object.keys(WidgetType).find(
    type => WidgetType[type].value === widget.type
  ),
  params: widget.params.map(paramId =>
    Object.keys(WidgetParams).find(type => WidgetParams[type].value === paramId)
  ),
  useConnector: widget.useConnector,
}));

console.log(JSON.stringify(mappedOutput, null, 2));

And you'll get the following output:

Decoded payload

[
  {
    "type": "ais.index",
    "params": [
      "indexName"
    ],
    "useConnector": false
  },
  {
    "type": "ais.refinementList",
    "params": [
      "container",
      "attribute",
      "searchable",
      "searchablePlaceholder",
      "searchableShowReset",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.hierarchicalMenu",
    "params": [
      "container",
      "attributes"
    ],
    "useConnector": false
  },
  {
    "type": "ais.clearRefinements",
    "params": [
      "container",
      "templates"
    ],
    "useConnector": true
  },
  {
    "type": "ais.configure",
    "params": [
      "attributesToSnippet",
      "snippetEllipsisText",
      "removeWordsIfNoResults"
    ],
    "useConnector": false
  },
  {
    "type": "ais.toggleRefinement",
    "params": [
      "container",
      "attribute",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.hitsPerPage",
    "params": [
      "container",
      "items"
    ],
    "useConnector": false
  },
  {
    "type": "ais.pagination",
    "params": [
      "container",
      "scrollTo",
      "padding",
      "showFirst",
      "showLast",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.rangeSlider",
    "params": [
      "container",
      "attribute",
      "pips",
      "tooltips"
    ],
    "useConnector": false
  },
  {
    "type": "ais.hits",
    "params": [
      "container",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.ratingMenu",
    "params": [
      "container",
      "attribute",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.stats",
    "params": [
      "container",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.searchBox",
    "params": [
      "container",
      "placeholder",
      "templates"
    ],
    "useConnector": false
  },
  {
    "type": "ais.sortBy",
    "params": [
      "container",
      "attribute",
      "items"
    ],
    "useConnector": false
  }
]

Haroenv · 2020-07-22T12:43:13Z

HTTP header

We shouldn't use a header, since that unfortunately changes the network request from a "simple request" (which doesn't need to comply with CORS), to a regular request, and thus needs a synchronous OPTIONS request (which makes the initial search slower)

Haroenv · 2020-07-22T12:52:10Z

schema.js

-  pips: 1040,
-  tooltips: 1041,
-});
+var WidgetParams = exports.WidgetParams = {


would there be a way to make this esm instead of commonjs?

Sadly pbf only output a commonjs module (issue: mapbox/pbf#97)

I think if you change these lines it should work; since they have an issue, they will probably accept the PR

https://github.com/mapbox/pbf/blob/master/compile.js#L94-L97

yannickcr · 2020-07-22T12:57:34Z

We shouldn't use a header, since that unfortunately changes the network request from a "simple request" (which doesn't need to comply with CORS), to a regular request, and thus needs a synchronous OPTIONS request (which makes the initial search slower)

Oh. You're right. We cannot use headers here then. So we need to keep using the query parameters.

POC: Add telemetry

0a9937a

Haroenv reviewed Jul 21, 2020

View reviewed changes

src/widgets/configure-related-items/configure-related-items.ts Outdated Show resolved Hide resolved

Update src/widgets/configure-related-items/configure-related-items.ts

c93f75f

Co-authored-by: Haroen Viaene <hello@haroen.me>

Haroenv reviewed Jul 21, 2020

View reviewed changes

Update src/widgets/index/index.ts

f5f45f9

Co-authored-by: Haroen Viaene <hello@haroen.me>

Use full Schema

522df84

Haroenv reviewed Jul 22, 2020

View reviewed changes

hc0503 approved these changes Nov 4, 2020

View reviewed changes

Haroenv closed this Jan 19, 2021

Haroenv deleted the poc/telemetry branch January 19, 2021 12:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

POC: Add telemetry #4452

POC: Add telemetry #4452

yannickcr commented Jul 21, 2020 •

edited

Loading

codesandbox-ci bot commented Jul 21, 2020 •

edited

Loading

Haroenv left a comment

Haroenv Jul 21, 2020

yannickcr Jul 22, 2020

Haroenv Jul 21, 2020

yannickcr Jul 22, 2020

Haroenv Jul 21, 2020

yannickcr Jul 22, 2020

Haroenv commented Jul 21, 2020 •

edited

Loading

Haroenv Jul 21, 2020

yannickcr Jul 22, 2020

Haroenv Jul 22, 2020

Haroenv commented Jul 21, 2020

yannickcr commented Jul 22, 2020 •

edited

Loading

Haroenv commented Jul 22, 2020

Haroenv Jul 22, 2020

yannickcr Jul 22, 2020

Haroenv Jul 23, 2020

yannickcr commented Jul 22, 2020

POC: Add telemetry #4452

POC: Add telemetry #4452

Conversation

yannickcr commented Jul 21, 2020 • edited Loading

codesandbox-ci bot commented Jul 21, 2020 • edited Loading

Haroenv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Haroenv commented Jul 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Haroenv commented Jul 21, 2020

yannickcr commented Jul 22, 2020 • edited Loading

Haroenv commented Jul 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yannickcr commented Jul 22, 2020

yannickcr commented Jul 21, 2020 •

edited

Loading

codesandbox-ci bot commented Jul 21, 2020 •

edited

Loading

Haroenv commented Jul 21, 2020 •

edited

Loading

yannickcr commented Jul 22, 2020 •

edited

Loading