Addition TikTok as a Provider

[jaredrileysmith]

…ded TIKTOK16x16.png to src/assets/images/icons and updated icons.scss accordingly

[coveralls]

Coverage decreased (-0.006%) to 90.513% when pulling 8c08e2a on jaredrileysmith:master into 9387aad on MisterPhilip:master.

[jaredrileysmith]

@johndavidsimmons let me know if having a URL available where all the TikTok events can easily be observed would be useful for completing this review & I can publish the necessary pixels to a site I have control of.

[johndavidsimmons]

Hey sorry I was out of town, I will review some time this week.

[johndavidsimmons]

What is business.topbuzz.com? I see the tiktok provider class tracking requests from that URL, but the tiktok pixel is sending post requests to https://analytics.tiktok.com/api/v1/track

[jaredrileysmith]

@johndavidsimmons the TikTok events send a GET request with the same data to the business.topbuzz.com endpoint, but as query string parameters. It's unclear why this is the case, but it has been observed that the GET requests are typically sent first. If necessary I can refactor to use the POST requests that are sent to the analytics.tiktok.com endpoint.

[jaredrileysmith]

FWIW TopBuzz and TikTok are both owned by ByteDance.

[johndavidsimmons]

Looks like you might get 2 for 1 here. I would add tiktok and topbuzz as separate providers with the tiktok provider parsing the post data to sent to analytics.tiktok.com and the topbuzz provider parsing the query param data being sent to business.topbuzz.com

[MisterPhilip]

@jaredrileysmith - so if you're on a single site, it'll send it to both domains, or are these two separate tags? Can you send me a URL that has TikTok on it (you can send it privately to philip@omnibug.io if want to keep it out of the public eye).

If its the latter, I'm not sure it's worth creating two providers for - it looks like topbuzz was technically shut down.

[jaredrileysmith]

@MisterPhilip I'll email you the URL where all the TikTok standard events are configured to fire on the home page.

[jaredrileysmith]

It is worth noting that the network calls to both domains are initiated by the TikTok SDK

[MisterPhilip]

I'm a little conflicted here, since they are largely contain the same payloads and it seems like it's a single base code that is sending data to both locations. Given that TopBuzz seems to be going away, and TikTok's requests are better formatted, I think we should stick with just the TikTok request. @jaredrileysmith / @johndavidsimmons do you see any value in keeping both or do you agree that sticking with one would be best?

Example from TikTok:

{
    "event": "Purchase",
    "context": {
        "ad": {
            "ad_id": "",
            "req_id": "",
            "creative_id": "",
            "log_extra": "{}",
            "idc": "",
            "convert_id": "",
            "callback": ""
        },
        "user": {
            "user_id": "",
            "device_id": ""
        },
        "pixel": {
            "code": "YYYYYYYYYYYYYYYYYYYYY"
        },
        "page": {
            "url": "https://domain.example.com/",
            "referrer": "https://referrer.example.com"
        },
        "library": {
            "version": "0.0.24",
            "name": "pixel.js"
        }
    },
    "analytics_uniq_id": "xxxxxxxxxxxxxxxxxxxx",
    "timestamp": "2020-09-03T20:35:14.262Z"
}

Example from TopBuzz:

device_id: 
user_id: 
uid: 
ut: 
client_version: 
version_code: 
req_id: 
cid: 
site_id: 
ad_id: 
track_data: [{
    "event_type": "on_web_order",
    "event_pixel_id": "123456789123456789",
    "event_pixel_code": "YYYYYYYYYYYYYYYYYYYYY",
    "advertiser_id": "1111111111111111111",
    "data_type": 2,
    "analytics_uniq_id": "xxxxxxxxxxxxxxxxxxxx",
    "options": {},
    "log_extra": "{}",
    "os": "",
    "page_url": "https://domain.example.com/",
    "page_type": 0
}]
tt_bridge: 1111
tt_env: 1110
app_id: 
analytics_uniq_id: xxxxxxxxxxxxxxxxxxxx
source: webunion
sdk_version: 0.0.24
t: Thu Sep 03 2020 13:35:14 GMT-0700 (Mountain Standard Time)

[MisterPhilip]

Correct, both are sent from the same JS file:

            getHttpConvertUrl: function() {
                var e = X || G.getFromSessionStorage("umeng_idc");
                switch (e + "") {
                case "3":
                    return "//business-sg.topbuzz.com/2/wap/landing_tetris_log/";
                default:
                    return "//business.topbuzz.com/2/wap/landing_tetris_log/"
                }
            },
            getInterface3Url: function() {
                return "https://analytics.tiktok.com/api/v1/track"
            },

The regex would need to be updated to match the -sg version too if we include TopBuzz

[jaredrileysmith]

@MisterPhilip I don't see any value in having both & I agree that the POST requests are a little better formatted, etc. I also agree that it's probably best to use the POST requests, given the uncertain nature of TopBuzz, etc. Moving forward, should I delete this pull request & submit a new one once the changes are made? Apologies for the ignorance, this is my first-ever pull request.

[MisterPhilip]

@jaredrileysmith - sounds good. Let's stick with the TikTok domain then.

For your pull request, you're fine! Just make the updates to the file and commit to your master branch (like you did to create the file) and it'll automatically update this pull request 😄 Definitely appreciate the contribution!!

[jaredrileysmith]

@MisterPhilip thanks for the clarification, I should have the update ready some time this weekend.

[jaredrileysmith]

@MisterPhilip I'm having a little trouble getting started. I started with just the constructor in the TikTok provider class but am getting an error for the extension from the catch block of the parsePostData method in the BaseProvider class when I update the local version of the extension. I have provided the following images for clarity.

Any insight you can provide would be greatly appreciated, apologies if I've overlooked something apparent.

[MisterPhilip]

Hmm, I can't seem to replicate that error - and all of the requests I've seen have been POST data with JSON as the body. Can you look at the network requests to see what requests are sent and if there are any with empty bodies/malformed JSON objects?

Just to confirm, below is the same as what you have, correct?

class TikTokProvider extends BaseProvider {
    constructor() {
        super();
        this._key = "TIKTOK";
        this._pattern = /https:\/\/analytics\.tiktok\.com\/api\/v1\/track/;
        this._name = "TikTok";
        this._type = "marketing";
        this._keywords = ["TikTok"];
    }
}

[jaredrileysmith]

@MisterPhilip I cleared the error from the extension and refreshed after building again & the error is no longer present. I have gotten all the events to display as desired with the POST data and will be writing the requisite tests in the next couple of days to update this pull request. Thanks again for all your help.

[jaredrileysmith]

@MisterPhilip I am almost finished writing the test suite for this provider, but am having trouble with a test to ensure the provider returns the requestType. Below is the test, along with the error I get when I run $ yarn run test. I have found writting the test for POST requests a little more challenging than those for requests with query parameters, etc. and appreciate any direction you can provide.

test("TikTokProvider returns requestType", t => {
    let provider = new TikTokProvider(),
        url = "https://analytics.tiktok.com/api/v1/track",
        viewContentPostData = {"event":"ViewContent","context":{"ad":{"ad_id":"","req_id":"","creative_id":"","log_extra":"{}","idc":"","convert_id":"","callback":""},"user":{"user_id":"","device_id":""},"pixel":{"code":"BT24M8TQUU2IQ2BVG9NG"},"page":{"url":"https%3A%2F%2Fdigital-intelligence.myshopify.com%2F","referrer":""},"library":{"version":"0.0.24","name":"pixel.js"}},"analytics_uniq_id":"yhczscrde4u6oe9lpmuns","timestamp":"2020-09-04T11:22:12.959Z"},
        purchasePostData = {"event":"Purchase","context":{"ad":{"ad_id":"","req_id":"","creative_id":"","log_extra":"{}","idc":"","convert_id":"","callback":""},"user":{"user_id":"","device_id":""},"pixel":{"code":"BT24M8TQUU2IQ2BVG9NG"},"page":{"url":"https%3A%2F%2Fdigital-intelligence.myshopify.com%2F","referrer":""},"library":{"version":"0.0.24","name":"pixel.js"}},"analytics_uniq_id":"mtcrxyl78c2gtebpccznh","timestamp":"2020-09-04T11:22:12.959Z"},
        checkoutPostData = {"event":"Checkout","context":{"ad":{"ad_id":"","req_id":"","creative_id":"","log_extra":"{}","idc":"","convert_id":"","callback":""},"user":{"user_id":"","device_id":""},"pixel":{"code":"BT24M8TQUU2IQ2BVG9NG"},"page":{"url":"https%3A%2F%2Fdigital-intelligence.myshopify.com%2F","referrer":""},"library":{"version":"0.0.24","name":"pixel.js"}},"analytics_uniq_id":"ty67gzsy1disrvnmos4zx","timestamp":"2020-09-04T11:22:12.958Z"},
        viewContentParsed = provider.parseUrl(url, viewContentPostData),
        purchaseParsed = provider.parseUrl(url, purchasePostData),
        checkoutParsed = provider.parseUrl(url, checkoutPostData),
        viewContentResult = viewContentParsed.data.find((result) => {
            return result.key === "requestType";
        }),
        purchaseResult = purchaseParsed.data.find((result) => {
            return result.key === "requestType";
        }),
        checkoutResult = checkoutParsed.data.find((result) => {
            return result.key === "requestType";
        });

    t.is(typeof viewContentResult, "object");
    t.is(viewContentResult.value, "Event");

    t.is(typeof purchaseResult, "object");
    t.is(purchaseResult.value, "Event");

    t.is(typeof checkoutResult, "object");
    t.is(checkoutResult.value, "Event");
});

[MisterPhilip]

Wrap the POST data in a string, since it'll attempt to do a full parse on it. e.g.:

viewContentPostData = `{"event":"ViewContent","context":{"ad":{"ad_id":"","req_id":"","creative_id":"","log_extra":"{}","idc":"","convert_id":"","callback":""},"user":{"user_id":"","device_id":""},"pixel":{"code":"BT24M8TQUU2IQ2BVG9NG"},"page":{"url":"https%3A%2F%2Fdigital-intelligence.myshopify.com%2F","referrer":""},"library":{"version":"0.0.24","name":"pixel.js"}},"analytics_uniq_id":"yhczscrde4u6oe9lpmuns","timestamp":"2020-09-04T11:22:12.959Z"}`,

[jaredrileysmith]

@MisterPhilip I tried wrapping the POST data in a string & also tried to call JSON.stringify() on the objects (to create strings?) but neither was successful and the error is persistent. I will do some more research on the method listed for parsing POST data from Stack Overflow to see if I can get a better understanding of what is happening.

[MisterPhilip]

Hmm, actually I'm wrong - you can pass an object as POST data. (sorry!)

Are you returning a requestType parameter in your provider file? For example, the Adobe Target provider looks for the mboxVersion parameter, which is set in the handleCustom method.

results.push({
    "key":   "mboxType",
    "field": "Mbox Type",
    "value": matches[2],
    "group": "general"
});

It should be whatever the value of the requestType is in the columnMapping getter:

get columnMapping()
{
    return {
        "account":      "mbox",
        "requestType":  "mboxType"
    };
}

[jaredrileysmith]

@MisterPhilip thanks for the clarification. I am returning a requestType in the provider file, it looks like this

get columnMapping()
{
    return {
        "account":     "context.pixel.code",
        "requestType": "event"
    };
}

The requestType is the value of the first key in the POST data. I am not overriding the handleCustom method in my provider file. Could this be an issue with how the POST data is formatted / the object is nested? For reference here's an example POST payload (all the events have identically formatted payloads).

{
   "event":"ViewDownloadPage",
   "context":{
      "ad":{
         "ad_id":"",
         "req_id":"",
         "creative_id":"",
         "log_extra":"{}",
         "idc":"",
         "convert_id":"",
         "callback":""
      },
      "user":{
         "user_id":"",
         "device_id":""
      },
      "pixel":{
         "code":"BT24PR2UH19GN34U0RP0"
      },
      "page":{
         "url":"https%3A%2F%2Fdigital-intelligence.myshopify.com%2F",
         "referrer":""
      },
      "library":{
         "version":"0.0.24",
         "name":"pixel.js"
      }
   },
   "analytics_uniq_id":"jn1u8wjhjrcvtytdpi4toc",
   "timestamp":"2020-09-04T20:33:22.634Z"
}

I'll keep looking into the methods for the BaseProvider class to see if I can determine what the issue is. Thanks again for all your help.

[MisterPhilip]

Use event then:

        viewContentResult = viewContentParsed.data.find((result) => {
            return result.key === "event";
        }),
        purchaseResult = purchaseParsed.data.find((result) => {
            return result.key === "event";
        }),
        checkoutResult = checkoutParsed.data.find((result) => {
            return result.key === "event";
        });

[jaredrileysmith]

@MisterPhilip I accidentally created another pull request. Should I close this one out?