Get job runs by timeframe

[JayaChinta123]

Describe the feature

For glue service,get-job-runs method return all the job runs from the time the job is created and we have to filter the response basing on completed-on attribute. It will be nice to have get job runs accept time as an input parameter,so the response can be limited to fewer number of records.

Use Case

We need to calculate max concurrent dpu utilized in an account at some point in time, so basing on which we can calculate dpu concurrency and can request the quota for the account.
For that we had query glue service for job runs and it is returning all job runs from the time job is created. In an account where we more than thousand jobs and to retrieve job runs for each job from the time they are created ..the script takes a day to run!!! As the job get older the latency increases

Proposed Solution

Expose time as input parameter for glue Job runs. And remove if possible older job runs basing on some rention period.so we don’t get large response data to scan results.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

Boto1.28.30

Environment details (OS name and version, etc.)

Mac

[RyanFitzSimmonsAK]

Hi @JayaChinta123, thanks for submitting this feature request. Server-side filtering is something that would need to be implemented by the service team. I've made an issue in our cross-SDK repository (aws/aws-sdk#591) to be used for tracking and updates. I've also reached out to the Glue service team to ask about the possibility of this being implemented.

One possible workaround you could use in the meantime is using an alternative data store like DynamoDB, and updating each job run in there. You could then query DynamoDB instead of having to make frequent GetJobRun calls.