Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust sensor to better support large jira projects #46

Merged
merged 2 commits into from
Feb 13, 2023

Conversation

floatingstatic
Copy link
Collaborator

@floatingstatic floatingstatic commented Sep 9, 2022

Addresses issue described in #45

Fixes #45

@floatingstatic
Copy link
Collaborator Author

Hi Eugen (@armab) I wonder if I could bump this PR which addresses #45. Currently I have to use a forked version of this pack due to the size of some Jira projects in my org. The way the sensor currently works as I understand it is we load all issues in a jira project into the sensor object under self._issues_in_project then poll for new issues and check if that issue is in the list to determine if we should emit a trigger. With 30-40k issues this never seems to work for me due to the volume of data the jira api has to return. This change I am proposing simply loads the latest issue id with a minor modification to the jql at sensor startup and polls for issues newer than the last known id we recorded at startup. We then emit triggers for any issues newer than self._latest_id, and finally update self._latest_id after triggers are emitted. This seems more efficient and scales to very large Jira projects (and is working well for my use case with very large Jira projects). From the user's perspective this change should be a no-op (functionally the sensor produces the same result as the current logic [ie. we just diff new issues vs subsequent polls to emit triggers]).

If you are not the current maintainer of this pack perhaps you could pull in that person to help review this change? I'm also happy to help support this pack as I do for stackstorm-napalm if that is of any use to the st2 community.

Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff! 💯

JIRA is known to be used with lots of data/issues and manytimes things are slow.
This one is very helpful, thanks for the contribution!

Happy to add you as a maintainer @floatingstatic for this pack.
Could you please open a PR similar to this https://github.com/StackStorm-Exchange/stackstorm-napalm/pull/68/files in this repo to list yourself as a jira pack maintainer?

all_issues = self._jira_client.search_issues(self._jql_query, maxResults=None)
self._issues_in_project = {issue.key: issue for issue in all_issues}

self._jql_query = 'project={} ORDER BY id DESC'.format(self._project)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another addition would be storing the latest ID in the st2 K/V so if the sensor is restarted it will remember where it stopped last time.
https://docs.stackstorm.com/sensors.html#datastore-management-operations

Many pack Sensors are implemented this way, would it make sense for JIRA too? Maybe in the next PR, if you want to implement that?

@arm4b arm4b merged commit 283cc7f into StackStorm-Exchange:master Feb 13, 2023
@floatingstatic
Copy link
Collaborator Author

Hi @armab yeah i think using the kv store makes sense and I'm happy to add that feature in another PR (will open an issue to track this)

@floatingstatic
Copy link
Collaborator Author

Opened #51 for maintainer
Thanks for again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sensor times out searching for issues with very large JIRA project
2 participants