Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automate benchmarks via CML #518

Merged
merged 55 commits into from Nov 18, 2020
Merged

Automate benchmarks via CML #518

merged 55 commits into from Nov 18, 2020

Conversation

tholor
Copy link
Member

@tholor tholor commented Oct 23, 2020 •
edited

Automate our benchmarking to simplify and standardize runs

  • Configure AWS
  • Run CML Job via GitHub actions
  • Add additional services (postgres + elastic)
  • Adjust benchmark script for running services
  • Move to GPU
  • Add results as comment in PR
  • Fix reader accuracy metrics
  • Manual triggers / label-based triggers
  • Verify shutdown of machines in all cases

@lalitpagaria
Copy link
Contributor

lalitpagaria commented Oct 26, 2020 •
edited

Try this to find out which missing IAM permission error.
https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-encoded-authorization-failure/

Error creating machine: Error in driver during machine creation: Error launching instance: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: yVRTp2JEqKXxSgx48oFBKOTOJ_B7-WyYYKlUq3vsZqK9U6dO4SBd5Zjr5ofc6k_JOMxrM0sreGP8Kmt6gnHsMUGyRu-J2x1a-DjX-KtOoaRRIGlNayCGIeAUXoh08AgWA_gkKL2a5I9PuSCUbKgCu2AfZsDPLftbL4lGNmQF6lMoFkMpQGbUb_uUH7XIF95Vl0JFIKqPrZWXD4GOjhqyAEPeucvPzVEkYrbZTa-C5irTufRaQePZHBk8dPLHdtqrRfk_dpeDqjSJ5ui8eb1IpuAk9v0vjOfzHFYT_iiWmMjhDrer3KNBEw95dHjNCiM4XkkBmRXerC3Ntt4j0smYArIiV5P2pbe3mQlW14nZjDQJ_qc1L0q2DU-Fma_VmLAVexgp7FglmRvc_buM_ka0D4uH7rRahjlNFKiHV2bg3Ydnvi4a2En0i16SMigt4HmzmKQH0pmIqyVwH_QLP_CTAhrxLVu1OhYW1JSI7IbayhBd8jDogE2aEFQv4rYDwSRYxuA7GyDztkjHSSx-4HkT5hZYZMiwGGmLSMNO27VTZjX8uGPLC1Xr3mIVKEWyWgAjLwOmLzpqhQMWK9VBJ8JSzpsIUyCi-gwLesoS36z1TVXt21fi3i3RAeOEWBAWHeH4pWSbkp_Gb3p4TqlrENv5wbqHoqmNWaEHIfZVWupkdg

@tholor
Copy link
Member Author

tholor commented Oct 26, 2020

@lalitpagaria Yep, already on it...

@tholor tholor added this to the #4 milestone Nov 4, 2020
@tholor tholor changed the title WIP Automate benchmarks via CML Automate benchmarks via CML Nov 18, 2020
@tholor tholor merged commit 0acafc4 into master Nov 18, 2020
@tholor tholor deleted the add_cml_benchmarks branch November 18, 2020 17:28
@tholor tholor restored the add_cml_benchmarks branch November 27, 2020 10:35
@julian-risch julian-risch deleted the add_cml_benchmarks branch November 15, 2021 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lalitpagaria lalitpagaria lalitpagaria left review comments

Assignees

@tholor tholor

Labels
None yet
Projects
None yet
Milestone
#4
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@tholor @lalitpagaria @HaystackBot