create array of a string (liquid: split?!)

[hunhejj]

Hi,

I have a string that contains urls I'd like to feed the WebsiteAgent with.
How can I create an array out of a string? I've tried the liquid filter 'split', however the endresult is still a string. I even tried to create the url array with liquid directly in the websiteagent, unfortunately without any efforts.

Actually what I try to accomplish is generating multiple events based on a single html email fetched by the imap agent. As the website agent can't scrape an event payload (yet? see request for
WebsiteAgent to extract from event payload instead of url, I've collected all the links I'm interested in with an EventFormatter agent and now I want to scrape them one by one. So if you have any better solution, I'm interested!

Thanks in advance!
Hunhejj

[hunhejj]

I did a proof of concept for the other issue (issue 1050), that seems to work. But as I have no experience with git, I don't know how to make a push request (especially because I haven't done pulls recently).

[cantino]

Hey @hunhejj, it sounds like having the website Agent parse your HTML would be useful, but that you're also having trouble splitting into an array. What Liquid are you using.

[hunhejj]

let's say I have a string with spaces. I assumed if I use "myArray" = "{{myString | split: ' '}}" in the event formatting agent then I get an array.

But actually I modified the website agent so that it parses the HTML contained by a key of the event.

[cantino]

What actually comes out of "{{myString | split: ' '}}"?

[hunhejj]

myString

[cantino]

You're correct that when I pass

{
  "myString": "foo bar"
}

into an EventFormattingAgent that looks like

{
  "instructions": {
    "foo": "{{myString | split: ' '}}"
  },
  "mode": "clean"
}

I get

  {
    "foo": "foobar"
  }

@dsander, do you have any idea what we'd need to do in order to allow Liquid to output structured data as a result?

[dsander]

@cantino I don't know much about the internals of liquid, but since it is designed as a templating engine I would suspect that it is not designed to output anything else then strings. Maybe we could create a filter which takes the output of split and emits the returned springs as new events? Not sure if that would be too useful because it would be hard to get any additional information into that event.

@hunhejj You could use a javascript agent to split the string and emit new events for your WebsiteAgent

[ohingardail]

Liquid templating splits strings into arrays quite well, it's just that you have to do something with the array within the same Liquid template to notice, and the output is always a string (ie, the "array" only exists within the template).

Example: I want to get the last bit of a URL domain or subdomain :
{{ url | regex_replace: '/(index\\..*)?\\s*\\Z', '' | split: '/' | last }}

1. {{ url }} is something like "http://a/b/c/index.html"
2. {{ url | regex_replace: '/(index..*)?\\s*\\Z', '' }} is "http://a/b/c"
3. {{ url | regex_replace: '/(index..*)?\\s*\\Z', '' | split: '/' }} is an array ["http:","", "a, "b","c"] (an array which exists at this point only and cannot be exported to Huginn)
4. the Liquiid filter 'last' takes the last element "c" and emits it as a string.

If you want to do more complex stuff with an array (https://docs.shopify.com/themes/liquid-documentation/filters/array-filters) you still have to do it within the Liquid template. For example, I have a working website agent with this config to extract the main domain name of a URL:

"domain": "{% assign element = url | split: '/' %}{{ element[2] | regex_replace: '\\Awww\\.', '' | strip }}",

If you wanted to iterate through each element in an array, you can use Liquid's {% for ... %} loop
(https://docs.shopify.com/themes/liquid-documentation/tags/iteration-tags) :

"thingy": "{% for element in url %} {% if element == 'XXX' %} Whoopee! {% endif %}  {% endfor %}"

... or, depending on what you want to do and how much you know about your array, set up as many properties in the Huginn agent as you have elements and extract each one in turn, then process each property in a downstream agent :

"element0": "{% assign element = url | split: '/' %} {{ element[0] }}",
"element1": "{% assign element = url | split: '/' %} {{ element[1] }}",
"element2": "{% assign element = url | split: '/' %} {{ element[2] }}",
... etc

In the above 'foobar' example, a working config would be :

{
  "instructions": {
    "foo": "{{myString | split: ' ' | first}}",
    "bar": "{{myString | split: ' ' | last}}"
  },
  "mode": "clean"
}

[cantino]

Thanks @ohingardail, that's helpful!

[hunhejj]

Thanks @ohingardail, that was really very helpful!

Too bad, that the liquid arrays can't get converted to "real" arrays, which would allow a clean event propagation without dirty regex hacks, etc. especially in case I don't know the number of array elements. Anyway, thank you for the great collection of possibilities!

[cantino]

Hey @thelittlesaul, I believe that url_from_event can only match a single URL. You can't scrape multiple URLs from a single incoming event right now. You might need to use a JavaScriptAgent to split that string into multiple events for now.

[thelittlesaul]

Thanks @cantino for your advice, I finally got it works with the JavaScript agent!

[cantino]

Nice @thelittlesaul!

[johnatasjmo]

I tested this code for extracting domain

"domain": "{% assign element = url | split: '/' %}{{ element[2] | regex_replace: '\\Awww\\.', '' | strip }}",

and works well.

@ohingardail , what would be the code for extracting domain and display it without www? (considering that some domains are i.e. domain.com.uk)