Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud matchers #2473

Merged
merged 13 commits into from Aug 25, 2022
Merged

Cloud matchers #2473

merged 13 commits into from Aug 25, 2022

Conversation

rdettai
Copy link
Contributor

@rdettai rdettai commented Jul 29, 2022

Offer out of the box matchers from external datasources (Abuse Feodo Tracker, Pulsedive). The matches are published to an SQS queue by a matcher-client service running on fargate.

馃摑 Checklist

  • All user-facing changes have changelog entries.
  • The changes are reflected on vast.io, if necessary.
  • The PR description contains instructions for the reviewer, if necessary.

馃幆 Review Instructions

Start the review by checking out the new documentation

I have fixed a few issues I encountered along the development of the feature:

  • refactored the task lifecycle management into common.py as we have both a vast-server and a matcher-client task now
  • re-organized the way docker images are managed: they are now all pushed to the same ECR and we use the tag to distinguish which one is used by which service. This avoids needing to create new repositories for plugins that also need use images (which is the case of the matcher with its matcher client)
  • added a VASTCLOUD_TRACE flag to help debugging the CLI (by default the stack traces are hidden)
  • fixed the dashboard
  • increase the VAST server memory to avoid OOM crashes (most of the cost comes from the CPUs anyway)

@rdettai rdettai added the feature New functionality label Jul 29, 2022
@rdettai rdettai self-assigned this Jul 29, 2022
@rdettai rdettai changed the title Story/sc 35289/cloud matchers Cloud matchers Jul 29, 2022
@rdettai
Copy link
Contributor Author

rdettai commented Jul 29, 2022

TODO:

@rdettai rdettai force-pushed the story/sc-35289/cloud-matchers branch 6 times, most recently from 3a0125a to bc4d05c Compare August 13, 2022 08:25
@rdettai
Copy link
Contributor Author

rdettai commented Aug 13, 2022

@rdettai
Copy link
Contributor Author

rdettai commented Aug 15, 2022

@rdettai rdettai marked this pull request as ready for review August 16, 2022 10:21
@rdettai rdettai requested a review from dispanser August 16, 2022 10:21
@rdettai
Copy link
Contributor Author

rdettai commented Aug 16, 2022

@rdettai rdettai requested a review from KaanSK August 16, 2022 12:30
@rdettai rdettai force-pushed the story/sc-35289/cloud-matchers branch from c0f43a5 to d1685b4 Compare August 16, 2022 14:12
@rdettai rdettai mentioned this pull request Aug 18, 2022
2 tasks
@rdettai
Copy link
Contributor Author

rdettai commented Aug 18, 2022

@dispanser don't review the docs to harshly, I will actually bring another batch of improvements to it in #2510 馃槈

Copy link
Contributor

@dispanser dispanser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and was able to see matching events in the SQS queue. Thanks!

A remark and a spelling correction in the comments :)

web/docs/setup-vast/deploy/aws-pro.md Outdated Show resolved Hide resolved
cloud/aws/resources/scripts/matcher-client.py Show resolved Hide resolved
@rdettai rdettai force-pushed the story/sc-35289/cloud-matchers branch from 9f29769 to 82de3b8 Compare August 24, 2022 08:38
@rdettai rdettai enabled auto-merge August 24, 2022 08:38
@rdettai rdettai force-pushed the story/sc-35289/cloud-matchers branch from 82de3b8 to e6fb8f2 Compare August 25, 2022 06:57
@rdettai rdettai merged commit a6afd23 into master Aug 25, 2022
@rdettai rdettai deleted the story/sc-35289/cloud-matchers branch August 25, 2022 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality
Projects
None yet
2 participants