Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create maintenance activity ETL workflow #939

Merged
merged 69 commits into from
Apr 26, 2023
Merged

Create maintenance activity ETL workflow #939

merged 69 commits into from
Apr 26, 2023

Conversation

klai95
Copy link
Contributor

@klai95 klai95 commented Mar 10, 2023 •
edited
Loading

@klai95 klai95 self-assigned this Mar 10, 2023
@klai95 klai95 marked this pull request as draft March 10, 2023 21:23
@klai95 klai95 mentioned this pull request Mar 10, 2023
Closed
@codemonkey800 codemonkey800 added the new-feature Release Label: Used for categorizing features in automated release notes label Mar 16, 2023
@klai95
added the maintenance activity etl model in model.py
686b478
@klai95
added the MaintenanceActivity class in model.py
3a713e8
@klai95
refactored naming of classes to GitHub in model.py
12c2537
@klai95
updated table name in model.py
4a9fba6
@klai95
added logic to update github activity to handler.py
c57c671
@klai95
added the update_github_activity method to handle updates of github d…
5ace12f 
…ata with respect to dyanamodb
@klai95
added _write_activity_to_dynamo method for github etl
4aa3480
@klai95
refactored comment in update_github_activity
f6740ef
@klai95
added a new method in the GitHubActivity class
121fe70
@klai95
deleted handler.py since it will exist after Manasa's PR is merged in
f19001b
@klai95
added more details to class and method
22af835
@klai95
update class GitHubActivityType values for variables: LATEST, MONTH, …
685b5fd 
…TOTAL
@klai95
added methods to github_activity_model.py
129c0af
@klai95
added update_github_activity method to processor.py
45429e8
@klai95
added code related to github data on handler.py and processor.py
b5a4973
@klai95
modified format for LATEST data store
d7a50fd
@klai95
import github_activity_model to be used in snowflake_adapter.py
76311f7
@klai95
added get_plugins_with_commits_in_window method and called the method…
6d18de5 
… in update_github_activity method
@klai95
modified class GitHubActivityType to reflect the schema of the mainte…
8bcfb60 
…nance activity ETL workflow
@klai95
update the get_query_timestamp_projection method
b6e90f0
@klai95
pushed code for debugging
4f2fdc7
@klai95
added mode sql code for fetching github commits and updated the model
0bd02ef
manasaV3
manasaV3 reviewed Apr 12, 2023
View reviewed changes
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
.happy/terraform/modules/ecs-stack/main.tf Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
manasaV3
manasaV3 previously requested changes Apr 12, 2023
View reviewed changes
data-workflows/activity/snowflake_adapter.py Outdated
Comment on lines 50 to 51
GROUP BY plugin, timestamp
ORDER BY plugin, timestamp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice clean up magic. ✨

data-workflows/activity/snowflake_adapter.py Outdated
Comment on lines 78 to 83
return _mapped_query_results(
query=github_activity_type.get_query(plugins_by_earliest_ts),
schema="GITHUB",
accumulator={},
accumulator_updater=github_activity_type.get_accumulator_updater(),
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very clean and easy to understand. 🔥

data-workflows/activity/snowflake_adapter.py Outdated Show resolved Hide resolved
data-workflows/handler.py Outdated Show resolved Hide resolved
data-workflows/handler.py Outdated Show resolved Hide resolved
data-workflows/handler.py Outdated Show resolved Hide resolved
codemonkey800
codemonkey800 previously approved these changes Apr 12, 2023
View reviewed changes
Copy link
Collaborator

@codemonkey800 codemonkey800 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one small comment but LGTM

data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
@klai95 klai95 requested a review from edu-bob April 13, 2023 17:21
@klai95
converted plugin_name to lower case to be stored in dynamoDB to maint…
6ddf071 
…ain parity with the implementation of install activity
@klai95
refactored helpers.py and github_activity_model.py such that the dict…
e7cdafe 
…ionary contains lower case value of plugin name, and therefore there is no need for plugin_name.lower()
kne42
kne42 reviewed Apr 14, 2023
View reviewed changes
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/snowflake_adapter.py Outdated Show resolved Hide resolved
data-workflows/activity/snowflake_adapter.py Show resolved Hide resolved
kne42
kne42 previously approved these changes Apr 14, 2023
View reviewed changes
Copy link
Member

@kne42 kne42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots cleaner thanks!

codemonkey800
codemonkey800 previously approved these changes Apr 14, 2023
View reviewed changes
Copy link
Collaborator

@codemonkey800 codemonkey800 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@klai95 klai95 dismissed stale reviews from codemonkey800 and kne42 via dd71781 April 24, 2023 21:07
kne42
kne42 previously approved these changes Apr 24, 2023
View reviewed changes
Copy link
Member

@kne42 kne42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some nits, otherwise lgtm

data-workflows/activity/github_activity_model.py Outdated Show resolved Hide resolved
data-workflows/activity/snowflake_adapter.py Outdated Show resolved Hide resolved
@klai95 klai95 enabled auto-merge (squash) April 26, 2023 18:21
@klai95 klai95 dismissed stale reviews from richaagarwal and manasaV3 April 26, 2023 18:22

feedback has been addressed

@klai95 klai95 merged commit e036ab2 into main Apr 26, 2023
@klai95 klai95 deleted the maintenance-etl-workflow branch April 26, 2023 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@codemonkey800 codemonkey800 codemonkey800 approved these changes

@richaagarwal richaagarwal richaagarwal left review comments

@manasaV3 manasaV3 manasaV3 left review comments

@kne42 kne42 kne42 left review comments

Assignees

@klai95 klai95

Labels
new-feature Release Label: Used for categorizing features in automated release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@klai95 @manasaV3 @richaagarwal @codemonkey800 @kne42