feat: Opsgenie plugin implementation #6154

sandesvitor · 2023-09-26T21:04:18Z

Summary

Adds a new plugin for Atlassian Opsgenie. Also includes Opsgenie Dashboard for Grafana.

Does this close any open issues?

Closes #6055

Screenshots

Include any relevant screenshots here.

Other Information

Any other information that is important to this PR.

klesh · 2023-09-27T02:00:32Z

Amazing, config-ui and dashboards are also included, very impressive work. 👍

@keon94 Would you like to help review this PR? thanks.
@mintsweet Please help review the config-ui part
@Startrekzky Please take a look at the dashboard, thanks

sandesvitor · 2023-09-27T13:51:43Z

Hey @klesh, thanks for the response! I also submitted this PRs to complement Opsgenie DOCs reference in config-ui.

apache/incubator-devlake-website#649

Posting here because I think they are related

keon94 · 2023-09-28T23:56:21Z

backend/plugins/opsgenie/tasks/incidents_collector.go

+		return err
+	}
+
+	err = collectorWithState.InitCollector(api.ApiCollectorArgs{


Does their API not support incremental collections? Meaning, to query based on incident creation time? Ideally, that's what we want otherwise we'll be pulling everything each time the blueprint runs.

Thanks for the feedback @keon94 ! So, I was testing the use for api.NewStatefulApiCollectorForFinalizableEntity. I just made a commit using this approach in the file backend/plugins/opsgenie/tasks/incidents_collector.go . With that said, I could not implement CollectUnfinishedDetails to fetch individual incidents that have different status then "closed" or "resolved". When trying to use CollectUnfinishedDetails the raw table is created but never populated =(

It is mandatory to use CollectUnfinishedDetails if we want to query based on incident creation time?

Just a additional information, Opsgenie API falls in the case of "Undetermind Strategy", because you can sort by Created Date in Descending order.

CollectUnfinishedDetails is something that runs when you already have existing unresolved incidents from the previous run. The purpose of it is to grab those incidents from the DB and check them against the API and pull down their newest state. It is pretty important to implement this otherwise the collected incidents will remain "stuck" in whatever state they were initially collected with and never get updated.
Have you looked at the PagerDuty implementation of it as reference?

incubator-devlake/backend/plugins/pagerduty/tasks/incidents_collector.go

Line 111 in 62a1e93

CollectUnfinishedDetails: &api.FinalizableApiCollectorDetailArgs{

I think @keon94 is proposing to use another helper
https://github.com/apache/incubator-devlake/blob/62a1e93a2dab358af6b0e2b1eecd21ad70208941/backend/plugins/pagerduty/tasks/incidents_collector.go#L69C24-L69C67
and I believe he is right unless the Opsgenie API supports filtering by updatedAt

Got it! I doing some tests, it appears and I use CollectUnfinishedDetails the incident collector first run does not populate the raw table _raw_opsgenie_incidents. But when I run again it does. I admit that I don't understand why hehe

Yep, that's expected behavior because it runs only if there's existing data in the table based on the output of the BuildInputIterator function.

It appears to be working now heheh

keon94 · 2023-09-29T00:09:54Z

backend/plugins/opsgenie/tasks/users_convertor.go

+		InputRowType: reflect.TypeOf(models.User{}),
+		Input:        cursor,
+		Convert: func(inputRow interface{}) ([]interface{}, errors.Error) {
+			user := inputRow.(*models.User)


are teams and users necessary to collect for the purpose of Dora metrics? Or is this plugin doing more than that?
cc @klesh

Hmm, you are right! In the context of DORA metrics I don't think they are necessary. My rational was to collect teams and usersin order to populate the responders table and be able to relate the issue with them and create the issue_assignee table (because an Opsgenie Incident does not have an assignee for it, only responders. It is a bit different from the concept of Incident in Pagerduty).

But, if you think it is overextending the needs for the plugin I'll be happy to remove them =)

I see. In that case, it is ok. Let's keep this.

mintsweet

LGTM about config-ui.

keon94 · 2023-10-06T01:33:19Z

backend/plugins/opsgenie/tasks/incidents_collector.go

+				Query: func(reqData *api.RequestData, createdAfter *time.Time) (url.Values, errors.Error) {
+					query := url.Values{}
+
+					query.Set("query", fmt.Sprintf("impactedServices:%s", data.Options.ServiceId))


You should be able to append the "createdAt" query to the query variable and remove the GetCreated function on L89. See the doc here:
https://support.atlassian.com/opsgenie/docs/search-syntax-for-incidents/

According to the doc, I agree with @keon94 and we should be using the same collection helper as pagerduty plugin.

Oh! Something like this?

I'm opened a ticket in Opsgenie to try and understand how this query works (the above example is not working yet)

It appears that I cannot test this...

Do you mean you need to be in an upgraded account to make that query? Is your token on a free plan?

What happens when you call the endpoint with this query right now? Anything in the response that says something like "query unsupported"?

It appear so, but I tested with a trail based on this Essential plan with the same result =(

It returns status code 400

Ok. Here's what I suggest then. Change the collection code back to how you had it first (full collection with no state), and put the state based logic in another branch. Then we'll merge this branch without that logic (please address the failing github checks first).
Go ahead and create another issue and call it "Support incremental collection for Opsengie plugin" and link that branch you created to it.

We'll worry about the performance enhancement in that ticket

I just pushed a change implementing the CollectUnfinishedDetails: &api.FinalizableApiCollectorDetailArgs{ and it is working properly. I just updated my branch and made a few changes to remote plugin api (adhering to the way Pagerduty plugin is doing like you oriented me).

That said, I used the GetCreated function.

Startrekzky

I reviewed the dashboard part in the JSON format. LGTM

klesh · 2023-10-16T02:08:47Z

We had code-frozen for v0.20, this PR would be merged on v0.21, with an ETA of the end of this month.
@keon94 Please approve the PR when you think it is ready.

keon94 · 2023-10-16T19:13:46Z

@klesh I've approved it. I think the plugin code looks ok. Please re-review and/or merge when you see fit.

klesh · 2023-10-17T11:03:49Z

@keon94 @sandesvitor Nice work, folks, will look into it as soon as v0.20 is released

klesh · 2023-10-17T11:11:45Z

@sandesvitor Seems like the config-ui failed, would you like to fix them as well? thanks.

sandesvitor · 2023-10-17T13:52:50Z

Thanks @klesh! Another question, could I commit ammend to fix one of my previous commit messages (that is breaking the lint)? I'm just a little bit confuse on how to write the command heheh

Something like this:

git commit --amend -m "fix(check): commit message"

klesh · 2023-10-18T06:54:47Z

@sandesvitor i believe what you need is https://jacopretorius.net/2013/05/amend-multiple-commit-messages-with-git.html

* fix: normalize_bp_settings throws error * fix: pg ci

d4x1 · 2023-10-20T07:08:14Z

@sandesvitor please help to resolve the lint errors on config-ui.

…or-devlake into feat-opsgenie-plugin

sandesvitor · 2023-10-24T17:26:40Z

Done @d4x1 ! =)

sandesvitor · 2023-11-07T13:37:36Z

Ah @klesh I forgot about the logo heheh the logo I used in config-ui is just a placeholder. It is to update in this PR or we have to wait?

klesh · 2023-11-08T04:27:58Z

@sandesvitor No problem, you may update the PR

sandesvitor · 2023-11-08T18:09:10Z

Got it @klesh ! But there is a any way to get the Opsgenie logo in Devlake standart? (I'm not a designer hahha)

klesh · 2023-11-09T12:25:19Z

@Startrekzky Please take a look.

Startrekzky · 2023-11-09T14:48:15Z

Hi @sandesvitor , I'll take care of the logo. I'll send it to you when it's done.

Startrekzky · 2023-11-10T08:18:04Z

Hi @sandesvitor , this is the logo in SVG format. Please check it out.

…or-devlake into feat-opsgenie-plugin

sandesvitor · 2023-11-10T19:21:46Z

Very nice @Startrekzky ! =) Works perfectly!

Startrekzky · 2023-11-15T14:22:01Z

Hi @klesh , since the v0.20 branch has been checked out, we can merge this PR now. Cc. @sandesvitor

Startrekzky · 2024-01-14T19:29:48Z

Hi @sandesvitor , I would like to nominate you as the Apache Committer. Your exceptional contributions to the Opsgenie plugin make you an ideal candidate for this. As an Apache Committer, you'll have the opportunity to make direct code contributions with push permission and collaborate closely with the community. If interested, please let me know, and I'll guide you through the process.

You can DM me (@louis Zhang) in the Slack community as well.

klesh requested a review from keon94 September 27, 2023 01:57

klesh requested review from mintsweet and Startrekzky September 27, 2023 02:00

keon94 reviewed Sep 28, 2023

View reviewed changes

keon94 reviewed Sep 29, 2023

View reviewed changes

mintsweet previously approved these changes Oct 3, 2023

View reviewed changes

keon94 reviewed Oct 6, 2023

View reviewed changes

Startrekzky previously approved these changes Oct 8, 2023

View reviewed changes

sandesvitor dismissed stale reviews from mintsweet and Startrekzky via 9dddf27 October 16, 2023 18:30

keon94 previously approved these changes Oct 16, 2023

View reviewed changes

sandesvitor dismissed keon94’s stale review via 615b510 October 16, 2023 19:22

sandesvitor mentioned this pull request Oct 17, 2023

feat: adding Opsgenie plugin DOCS apache/incubator-devlake-website#649

Merged

2 tasks

sandesvitor and others added 9 commits October 19, 2023 14:21

feat: Opsgenie plugin implementation

0332b5f

fix(lint): lint with gofmt

83758b9

refactor(lint): removing unused config-ui files

9fb9d35

feat(task): changed incident collector to accept incremental collection

1b17cf0

fix(config-ui): adjust the params for bp add connection (apache#6241)

3065bfa

refactor: removed unused code (apache#6243)

dff0a6f

refactor(config-ui): remove content about user info (apache#6245)

067e89a

fix(azure): fix 400 error when cloing azure repos (apache#6246)

82b0422

fix: normalize_bp_settings throws error (apache#6249)

af225fc

* fix: normalize_bp_settings throws error * fix: pg ci

sandesvitor and others added 5 commits October 20, 2023 09:48

fix(config-ui): removed old plugin type

11d0644

Merge branch 'feat-opsgenie-plugin' of github.com:sandesvitor/incubat…

b471516

…or-devlake into feat-opsgenie-plugin

Merge branch 'main' into feat-opsgenie-plugin

afdcf08

Merge branch 'main' into feat-opsgenie-plugin

77e1f71

Merge branch 'main' into feat-opsgenie-plugin

edd727f

sandesvitor added 2 commits October 26, 2023 09:54

Merge branch 'main' into feat-opsgenie-plugin

ddf11d2

refactor(config-ui): remove type inheritance from opsgenieconfig

7adbe60

klesh mentioned this pull request Oct 27, 2023

[Feature][Module Name] OpsGenie Plugin #6347

Closed

3 tasks

Merge branch 'main' into feat-opsgenie-plugin

f00e4f9

Merge branch 'main' into feat-opsgenie-plugin

2454632

sandesvitor and others added 4 commits November 10, 2023 16:02

Merge branch 'main' into feat-opsgenie-plugin

4e87f73

style(config-ui): update icon with icon.svg

94af888

Merge branch 'main' into feat-opsgenie-plugin

2950f6b

Merge branch 'feat-opsgenie-plugin' of github.com:sandesvitor/incubat…

bad3490

…or-devlake into feat-opsgenie-plugin

abeizn requested review from abeizn and d4x1 November 16, 2023 02:04

abeizn approved these changes Nov 24, 2023

View reviewed changes

abeizn merged commit 4453aa5 into apache:main Nov 24, 2023
11 checks passed

mintsweet mentioned this pull request Nov 24, 2023

fix(config-ui): adjust the code before pr merged #6512

Merged

feat: Opsgenie plugin implementation #6154

feat: Opsgenie plugin implementation #6154

Conversation

sandesvitor commented Sep 26, 2023

Summary

Does this close any open issues?

Screenshots

Other Information

klesh commented Sep 27, 2023 • edited Loading

sandesvitor commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keon94 Oct 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mintsweet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandesvitor Oct 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Startrekzky left a comment

Choose a reason for hiding this comment

klesh commented Oct 16, 2023

keon94 commented Oct 16, 2023

klesh commented Oct 17, 2023

klesh commented Oct 17, 2023

sandesvitor commented Oct 17, 2023

klesh commented Oct 18, 2023

d4x1 commented Oct 20, 2023

sandesvitor commented Oct 24, 2023

sandesvitor commented Nov 7, 2023

klesh commented Nov 8, 2023

sandesvitor commented Nov 8, 2023

klesh commented Nov 9, 2023

Startrekzky commented Nov 9, 2023

Startrekzky commented Nov 10, 2023

sandesvitor commented Nov 10, 2023

Startrekzky commented Nov 15, 2023

Startrekzky commented Jan 14, 2024

klesh commented Sep 27, 2023 •

edited

Loading

keon94 Oct 15, 2023 •

edited

Loading

sandesvitor Oct 10, 2023 •

edited

Loading