[Atlassian JIRA & Confluence] Add support for Atlassian Cloud #2715

legoguy1000 · 2022-02-19T23:01:31Z

What does this PR do?

Adds support for the SaaS versions of Confluence and JIRA in the Atlassian Cloud for the audit datastream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Integrations] Atlassian Jira/Confluence Cloud Integration #2695

Screenshots

elasticmachine · 2022-02-19T23:06:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-27T23:19:25.590+0000
Duration: 15 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	18
Skipped	0
Total	18

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-02-20T17:41:40Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

legoguy1000 · 2022-02-20T17:54:12Z

found some bugs, put back to draft

legoguy1000 · 2022-02-20T18:09:14Z

I'm getting this issue [elastic_agent.filebeat][debug] template execution failed: template: :1:18: executing "" at <.url.params.offset>: can't evaluate field params in type interface {}. Is this related to elastic/beats#30477 or elastic/beats#30476?? Also getting [elastic_agent.filebeat][error] error processing response: template: :1:29: executing "" at <.last_response.url.params.Get>: map has no entry for key "url" using a different agent config.

legoguy1000 · 2022-02-20T19:01:27Z

I was able to get around the above issue but now with the current config whenever it paginates, instead of just updating the offset via the response.pagination config, it also updates the date params using the cursor setting so it never actually paginates.

andrewkroh · 2022-02-24T20:35:48Z

I rebased this in order to apply the formatting of the pipeline test files in separate commits. This way it will be easier to digest the non-formatting changes during review. The JSON is now normalized due to elastic/elastic-package#644, but until all of the packages are normalized you'll get these large diffs the first time you make changes.

andrewkroh · 2022-02-24T23:13:44Z

/test

legoguy1000 · 2022-02-26T19:10:09Z

even with the latest 8.0.0 docker image, I can't get past the error, #2715 (comment). Any thoughts?

andrewkroh · 2022-02-28T13:58:23Z

Do the tests not expose this problem? The CI checks are passing for this. It's using 7.16.0 for the tests. The 8.0 branch was missing a backport, and that has now been fixed, but not released.

legoguy1000 · 2022-02-28T14:22:39Z

Do the tests not expose this problem? The CI checks are passing for this. It's using 7.16.0 for the tests. The 8.0 branch was missing a backport, and that has now been fixed, but not released.

I have to keep playing with it as I'm testing it against an actual Atlassian cloud instance and I keep getting the error so idk if I have a bad docker image or something but I've cleared all my local images and still having issues.

andrewkroh · 2022-02-28T14:58:00Z

Try testing with either 7.17.0 or 8.1.0 (e.g. elastic-package stack up -d -v --version=7.17.0).

legoguy1000 · 2022-03-02T16:10:16Z

I'll try with 8.1. I didn't have any issues when I built filebeat from source on main branch using a basic input config to test. So it's definitely weird.

legoguy1000 · 2022-03-11T19:02:37Z

@andrewkroh 8.1 appears to work and i'm no longer getting an error that the key doesn't exist but still having an issue with Pagination. I currently have

response.pagination:
  - set:
      target: url.params.from
      value: '[[.last_response.url.params.Get "from"]]'
      fail_on_template_error: true
...

This should maintain the existing URL parameter so the pagination continues through the same time range. However it's being overridden by the cursor and the original request Transforms. Any thoughts on how to fix that?

cursor:
  last_timestamp:
    value: "[[.first_event.created]]"

request.transforms:
...
  - set:
      target: url.params.from
      value: "[[.cursor.last_timestamp]]"
      default: '[[formatDate (now (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05.999-0700"]]'
...

P1llus · 2022-03-14T09:00:42Z

@legoguy1000 In the current situation the request transforms run for each pagination, so I believe we might have to take another approach on this.

If date is the only possible value here, we might have to drop the pagination and simply let it catch up with a smaller interval. Unless there is some sort of range query, or other values we can use from response headers or body to paginate.

Let's catch up when you got time to discuss this! :)

legoguy1000 · 2022-03-14T21:07:40Z

@P1llus copy. I'm artificially inducing pagination in order to make sure there is a valid solution as idk how many events/time is normal for these services. Unfortunately this seems like the only way to paginate but I can check again to see if there is anything in the headers.

P1llus · 2022-03-22T11:59:52Z

@legoguy1000
I would say that if there is limitations in the API that we can't easily work around, then we should simply not paginate, and rather have a smaller interval.
We can always have a discussion over slack if you want, to try to find other ways around it, but if there is none then we should keep it more simple, to ensure that it works correctly.

legoguy1000 · 2022-03-22T17:48:22Z

@P1llus Looks like instead of trying to set individual params, setting url.value and build the entire query string seems to be a work around. Only issue is it doesn't URL encode the values so the TZ offset in the JIRA date params +0000 messes up the url. It looks like just leaving off the TZ part will work since all the dates are in UTC anyways. BUt the better solution would probably be to create a URL encode template function. Thoughts??

legoguy1000 · 2022-03-22T17:58:04Z

Just tested without the TZ offset and the url.value, worked great. It paginated through 9 pages, 10 items each for 76 events total and the start/end dates stayed the same. Just need an answer on the URL encode functionality and we can press to merge this.

The value should be based as a YAML string. Prior to this change the final policy contained which is not what is expected. request.ssl: - object Object

legoguy1000 · 2022-04-25T20:15:21Z

@andrewkroh @P1llus Can we rerun the tests to see where we are. I think it should be good now. Currently its just not using the Time Zone in the dates since its all in UTC but when 8.3 is released, we can use the URL encode template function I made for the httpjson input to add it back just to be safe.

andrewkroh · 2022-04-26T03:19:52Z

/test

andrewkroh · 2022-04-27T20:39:43Z

/test

elasticmachine · 2022-04-27T20:55:32Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`6/6`)	💚 2.857
Classes	100.0% (`6/6`)	💚 2.857
Methods	100.0% (`45/45`)	💚 11.905
Lines	98.822% (`839/849`)	👍 9.985
Conditionals	100.0% (`0/0`)	💚

andrewkroh · 2022-04-27T21:00:30Z

Does any of this depend on elastic/beats#30962? If so the kibana constrained will need change to ^8.3.0.

legoguy1000 · 2022-04-27T21:04:29Z

Long term it should be updated to use that function, but I was able to get around it. Instead of wrapping the datetime with +000 for timezone, since its always UTC, i just left the TZ offset from the URL and it seemed to work fine.

andrewkroh · 2022-04-27T22:52:59Z

/test

event.action event.category event.created event.kind event.original event.type

event.action event.category event.created event.kind event.original event.outcome event.type

andrewkroh · 2022-04-27T23:18:54Z

/test

andrewkroh added Integration:atlassian_jira Atlassian Jira enhancement New feature or request Integration:atlassian_confluence Atlassian Confluence Team:Security-External Integrations labels Feb 20, 2022

legoguy1000 marked this pull request as draft February 20, 2022 17:53

andrewkroh force-pushed the 2695-atlasian-cloud branch from 2fb214c to 27801a6 Compare February 24, 2022 20:29

andrewkroh force-pushed the 2695-atlasian-cloud branch 2 times, most recently from b49488b to c89e547 Compare February 24, 2022 23:09

legoguy1000 marked this pull request as ready for review March 22, 2022 17:58

legoguy1000 requested a review from a team as a code owner March 22, 2022 17:58

legoguy1000 mentioned this pull request Mar 22, 2022

[Filebeat] Add URL Encode template function for httpjson input elastic/beats#30962

Merged

6 tasks

legoguy1000 force-pushed the 2695-atlasian-cloud branch from a463197 to 2ad4c82 Compare April 5, 2022 16:06

andrewkroh and others added 6 commits April 25, 2022 20:07

confluence: fix yaml variable format in test variables

63a4d50

The value should be based as a YAML string. Prior to this change the final policy contained which is not what is expected. request.ssl: - object Object

jira: fix empty processors for !atlassian_cloud

93cf392

jira: fix yaml variable format in test variables

f39f3ac

The value should be based as a YAML string. Prior to this change the final policy contained which is not what is expected. request.ssl: - object Object

using url.value works

e2984b5

worked

fb3952f

update docs

1fcd66f

legoguy1000 force-pushed the 2695-atlasian-cloud branch from 2ad4c82 to 1fcd66f Compare April 25, 2022 20:07

update generated data after rebase

a6fd574

fix readme's

a3dfa93

fix issues with timestamps

7750c3d

andrewkroh added 8 commits April 27, 2022 19:03

atlassian_confluence: Add missing event mappings

148c1fb

event.action event.category event.created event.kind event.original event.type

atlassian_confluence: Update readme

38f5d99

Separate config for atlassian cloud

280f63f

Separate timestamp processing

55b0de1

atlassian_jira: add missing event field mappings

3e459a6

event.action event.category event.created event.kind event.original event.outcome event.type

atlassian_jira: update readme

badd165

atlassian_jira: convert event.id to string

6f17062

atlassian_jira: update pipeline descriptions

131d4b4

andrewkroh approved these changes Apr 28, 2022

View reviewed changes

andrewkroh merged commit 0fffa6a into elastic:main Apr 28, 2022

andrewkroh mentioned this pull request May 2, 2022

[production] Promote many packages from snapshot elastic/package-storage#4302

Merged

andrewkroh mentioned this pull request May 9, 2022

[production] Promote many packages from snapshot elastic/package-storage#4362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Atlassian JIRA & Confluence] Add support for Atlassian Cloud #2715

[Atlassian JIRA & Confluence] Add support for Atlassian Cloud #2715

legoguy1000 commented Feb 19, 2022 •

edited

Loading

elasticmachine commented Feb 19, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Feb 20, 2022

legoguy1000 commented Feb 20, 2022

legoguy1000 commented Feb 20, 2022 •

edited

Loading

legoguy1000 commented Feb 20, 2022

andrewkroh commented Feb 24, 2022

andrewkroh commented Feb 24, 2022

legoguy1000 commented Feb 26, 2022

andrewkroh commented Feb 28, 2022

legoguy1000 commented Feb 28, 2022

andrewkroh commented Feb 28, 2022

legoguy1000 commented Mar 2, 2022

legoguy1000 commented Mar 11, 2022

P1llus commented Mar 14, 2022

legoguy1000 commented Mar 14, 2022

P1llus commented Mar 22, 2022

legoguy1000 commented Mar 22, 2022

legoguy1000 commented Mar 22, 2022

legoguy1000 commented Apr 25, 2022 •

edited

Loading

andrewkroh commented Apr 26, 2022

andrewkroh commented Apr 27, 2022

elasticmachine commented Apr 27, 2022 •

edited

Loading

andrewkroh commented Apr 27, 2022

legoguy1000 commented Apr 27, 2022

andrewkroh commented Apr 27, 2022

andrewkroh commented Apr 27, 2022

[Atlassian JIRA & Confluence] Add support for Atlassian Cloud #2715

[Atlassian JIRA & Confluence] Add support for Atlassian Cloud #2715

Conversation

legoguy1000 commented Feb 19, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Feb 19, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Feb 20, 2022

legoguy1000 commented Feb 20, 2022

legoguy1000 commented Feb 20, 2022 • edited Loading

legoguy1000 commented Feb 20, 2022

andrewkroh commented Feb 24, 2022

andrewkroh commented Feb 24, 2022

legoguy1000 commented Feb 26, 2022

andrewkroh commented Feb 28, 2022

legoguy1000 commented Feb 28, 2022

andrewkroh commented Feb 28, 2022

legoguy1000 commented Mar 2, 2022

legoguy1000 commented Mar 11, 2022

P1llus commented Mar 14, 2022

legoguy1000 commented Mar 14, 2022

P1llus commented Mar 22, 2022

legoguy1000 commented Mar 22, 2022

legoguy1000 commented Mar 22, 2022

legoguy1000 commented Apr 25, 2022 • edited Loading

andrewkroh commented Apr 26, 2022

andrewkroh commented Apr 27, 2022

elasticmachine commented Apr 27, 2022 • edited Loading

🌐 Coverage report

andrewkroh commented Apr 27, 2022

legoguy1000 commented Apr 27, 2022

andrewkroh commented Apr 27, 2022

andrewkroh commented Apr 27, 2022

legoguy1000 commented Feb 19, 2022 •

edited

Loading

elasticmachine commented Feb 19, 2022 •

edited

Loading

legoguy1000 commented Feb 20, 2022 •

edited

Loading

legoguy1000 commented Apr 25, 2022 •

edited

Loading

elasticmachine commented Apr 27, 2022 •

edited

Loading