Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Facade collection update #2184

Merged
merged 15 commits into from Feb 22, 2023
Merged

Facade collection update #2184

merged 15 commits into from Feb 22, 2023

Conversation

IsaacMilarky
Copy link
Contributor

Description

  • Remove repo_status from repo table and add it to collection_status table
  • Make facade into a separate collection hook so that commit updates can be queried and jobs can be queued based on detected repo updates.
  • Update facade config to remove unneeded options like 'force updates' and 'force analysis'. Now options for updates and analysis just have a simple toggle.
  • Change facade queries to reflect schema change
  • Change repo status to only be used to get status of whether repo was cloned or not. If it wasn't cloned it has a status of 'new', if it was cloned it has a status of 'Update.' Other status's made sense when we had a worker-based structure to collection but doesn't make sense for a message queue. This previous function will be done as part of the celery scheduling.

Signed commits

  • Yes, I signed my commits.

Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
…y removed except for their use in determining the status of whether a repo was cloned

Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
@sgoggins
Copy link
Member

@IsaacMilarky : Does this make sense for me to test first?
Also, I think some docs about how the jobs are managed in facade would be helpful.

Copy link
Contributor

@ABrain7710 ABrain7710 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused with some of the changes. I thought we were going to have only the facade_status since the repo_status is only relating to Facade so it isn't really a repo_status. I'm now just confused as to why we need both. Couldn't we just have a facade_status that has values of ("Pending", "Update", "Failed Clone", "Success", "Errored")? I thought this is what we came to the other day.

@@ -0,0 +1,101 @@
"""change CollectionStatus table to keep track of facade independently

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@IsaacMilarky This should only include, the changes that are needed for this pr. I can change this if you would like

@IsaacMilarky
Copy link
Contributor Author

I'm a bit confused with some of the changes. I thought we were going to have only the facade_status since the repo_status is only relating to Facade so it isn't really a repo_status. I'm now just confused as to why we need both. Couldn't we just have a facade_status that has values of ("Pending", "Update", "Failed Clone", "Success", "Errored")? I thought this is what we came to the other day.

You're right I'll make those changes. I think I just had forgotten that when I was making the clean_collection_status method since that resets the collection status of interrupted tasks to pending and from that I forgot to remove the repo_status field since that keeps track of the status of the git repo

…redundant

Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
Signed-off-by: Andrew Brain <61482022+ABrain7710@users.noreply.github.com>
Signed-off-by: Andrew Brain <61482022+ABrain7710@users.noreply.github.com>
ABrain7710
ABrain7710 previously approved these changes Feb 22, 2023
Copy link
Contributor

@ABrain7710 ABrain7710 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a few small changes, then tested it and it appears to be working as expected. I will note that with these changes it is important to set the collection_interval to a smaller value since the secondary data won't be collected until the primary data finishes, and the next augur_collection_monitor task runs again.

Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
…is 300 times the collection interval which is a minute by default

Signed-off-by: Isaac Milarsky <imilarsky@gmail.com>
@IsaacMilarky IsaacMilarky merged commit 2741c90 into dev Feb 22, 2023
@IsaacMilarky IsaacMilarky deleted the facade-collection-update branch February 22, 2023 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants