WebsiteAgent Extraction Problem accessing sub path data

[GChester]

I have been working on a problem for several weeks and could use some help.

I am using a WebsiteAgent to scrape data from Redmine to get our "In Progress" issues.

The problem is extracting the Custom Fields data in the "issue.custom_field" path.

below is a sample of an issue that is returned.

{
  "issue": {
    "id": 9107,
    "project": {
      "id": 180,
      "name": "Inbox"
    },
    "status": {
      "id": 2,
      "name": "In Progress"
    },
    "priority": {
      "id": 2,
      "name": "Normal"
    },
    "subject": "Revolutionizing Communication Through Video - Ziggeo - Ziggeo",
    "custom_fields": [
      {
        "id": 9,
        "name": "Website",
        "value": ""
      },
      {
        "id": 19,
        "name": "Links",
        "value": ""
      },
      {
        "id": 21,
        "name": "Chat Link",
        "value": "[Chat Room](http://zion.leadgenilluminati.com:8065/chat/channels/%id%)"
      }
    ],
    "created_on": "2020-06-15T16:36:55Z",
    "updated_on": "2020-06-16T13:18:29Z"
  }
}

I've tried sending this data to another Website Agent using the below payload but not able to extract the "custom fields" data successfully.

{
  "expected_update_period_in_days": "2",
  "data_from_event": "{{ issue.custom_fields | json }}",
  "type": "json",
  "mode": "all",
  "extract": {
    "id": {
      "path": "issue.custom_fields[*].id"
    }
  }
}

Through research, I have tried the following values for "data_from_event"

  "data_from_event": "{{ issue.custom_fields | replace: '=>', ':' }}",
  "data_from_event": "{{ issue.custom_fields | json }}",

Can anyone offer advice on how to extract these sub-paths into addressable values?

[dsander]

Hi!

I would suggest to either use the template feature of the WebsiteAgent (use that on the Agent that receives the data from the API) or a EventFormattingAgent like this:

{
  "instructions": {
    "issue_ids": "{{ issue.custom_fields | map: \"id\" | as_object }}"
  },
  "matchers": [],
  "mode": "merge"
}

[GChester]

@dsander Thanks a million for your suggestion. The template option was not as clear for me so I tested with the Event Formatting option and got the following results which is closer to what I'm trying to do.

"issue_ids": [
      9,
      19,
      21
    ]

However, in my example, I was trying to simplify and did not mention the other fields name & value in the object with the ID field.

How would you recommend using the MAP: for those fields as well?

I looked at the Liquid Template Map documentation and did not see any reference to using multiple fields.

I also tried this without success.

{{ issue.custom_fields | map: \"id\" \"name\" | as_object }}

What would be perfect is if I could get it in a format that I could address it as {{ Website }} which is the value of Name and the results would be the value of the value field.

So I guess that would be in the format;

Website:domain.com

Just to clarify that is;

(Name):(Value)

The ID in this case really is not very useful data.

[dsander]

A map functional generally transforms an object or a hash (same thing different name) into an array. Liquid is generally designed as a templating language that outputs text, we are kind of abusing it to transform data structures.

How do you expect the final output to look? Generating a string/text from all those custom_fields combine is pretty easy, getting an array of name:value combination might require a JavaScriptAgent

[GChester]

I made a condensed version of the JSON to explain what I'm trying to do;

{
  "issue": {
    "id": 1234,
    "subject": "This is a Task",
    "custom_fields": [
      {
        "id": 9,
        "name": "Website",
        "value": "www.mydomain.com"
      }
}

I can already access the normal fields {{ issue.id }} and {{ issue.subject }} to pass the data to the downstream agents, but I do not know how to access the custom fields.

In this example, I would like to be able to flatten/transform the custom_field called Website into {{ issue.Website }} and the data would be "www.mydomain.com".

Does that make sense?

[dsander]

Gotcha, this should work in a EventFormattingAgent:

{
  "instructions": {
    "website": "{% assign website = issue.custom_fields | | where: \"name\", \"Website\" | first %}{{website.value}}"
  },
  "mode": "merge"
}

I don't think it is possible to merge the website into the issue object though.

[GChester]

Ok, I'll try it. So basically it would be addressable as {{Website}} right?. That should work.

I'll report back.

Thanks again for all your help!

[dsander]

Almost, {{website}}, pretty sure capitalization matters.

[GChester]

@dsander

Almost, {{website}}, pretty sure capitalization matters.

Yes, sorry I was getting my references mixed up.

I have tested this and was not able to get it to work.

The value is coming up blank.

Any suggestions?

[dsander]

Without any additional information? No