Pre commit hooks (#18)

* Update CDK * Upgrade CDK * Add Dockerfile for ECS Agent * Add base bakery stack to deploy a cluster with 1 ECS agent running * Add instructions on creating RUNNER TOKEN secret pre-deployment * Upgrade prefect, add example flow * Refactor out permissions for ecs tasks into a specific role * Add ECS Run as we need to specify the image ECS users * Begin to add DaskExecutor to Flow - Need to build a image with Dask deps * Split out agent and worker docker images * Import container from ecr repository that is pre-populated * Migrate ECSRun to use dynamic task definition ref PrefectHQ/prefect/issues/4243 * Install dev deps via make due to pipenv locking bugs. Temporary hack. * Include dask-cloudprovider dependencies in worker image. * Stack role and export updates to support a DaskExecutor in test Flow. * Inclued agent label environment variable for deployment. * Initial test flow to validate DaskExecutor functionality. * Add additional bucket for processing cache and target output. * Add full zarr transform flow and move test flows to flow_test directory. * Include deps for transform_flow to avoid pipenv locking issue. * Add necessary dependencies for transform_flow execution by Dask workers. * Move test flows to flow_test. * Include local path for importing Flow tasks. * Fix linting issues. * Consolidate stack output value retrieval. * Move dev dependencies into Pipfile. * Propagate stack and flow tags to dynamically created ECS tasks. * Pin version of Github Actions ubuntu to support pipenv install. * Update directory paths for linting. * isort linting fixes. * Black formatting fixes. * Pin dependencies used by test flows. * Include detailed descriptions of new environment variables. * Update ids and stack exports to correctly use identifier in formatting. * Remove legacy comments from test flows and flow utils. * Add pre-commit hooks for linting and direct fixes. * Linting and formatting fixes to conform to pre-commit specs. Co-authored-by: Ciaran Evans <ciaran@developmentseed.org>
pangeo-forge · Apr 8, 2021 · 0a8596f · 0a8596f
1 parent b9a3f26
commit 0a8596f
Show file tree

Hide file tree

Showing 8 changed files with 182 additions and 39 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,53 @@
+default_stages: [commit]
+exclude: ^cdk.out/
+repos:
+  - repo: local
+    hooks:
+      - id: trailing-whitespace-fixer
+        name: trailing-whitespace-fixer
+        entry: pipenv run trailing-whitespace-fixer
+        language: system
+
+      - id: end-of-file-fixer
+        name: end-of-file-fixer
+        entry: pipenv run end-of-file-fixer
+        language: system
+
+      - id: check-docstring-first
+        name: check-docstring-first
+        entry: pipenv run check-docstring-first
+        types: [python]
+        language: system
+
+      - id: check-json
+        name: check-json
+        entry: pipenv run check-json
+        types: [json]
+        language: system
+
+      - id: check-yaml
+        name: check-yaml
+        entry: pipenv run check-yaml
+        types: [yaml]
+        language: system
+
+      - id: isort
+        name: isort
+        language: system
+        entry: pipenv run isort
+        types: [python]
+        args: ["--check-only", "--profile", "black", "cdk/", "flow_test/"]
+
+      - id: black
+        name: black
+        language: system
+        entry: pipenv run black
+        types: [python]
+        args: ["--check", "cdk/", "flow_test/", "--line-length", "100"]
+
+      - id: flake8
+        name: flake8
+        language: system
+        entry: pipenv run flake8
+        types: [python]
+        args: ["--max-line-length", "100", "cdk/", "flow_test/"]
diff --git a/Makefile b/Makefile
@@ -7,15 +7,14 @@ install:
 install-dev:
 	npm install
 	pipenv install --dev
+	pipenv run pre-commit install
 
 lint:
-	pipenv run flake8 cdk/ flow_test/
-	pipenv run isort --check-only --profile black cdk/ flow_test/
-	pipenv run black --check --diff cdk/ flow_test/
+	pipenv run pre-commit run --all-files
 
 format:
 	pipenv run isort --profile black cdk/ flow_test/
-	pipenv run black cdk/ flow_test/
+	pipenv run black cdk/ flow_test/ --line-length 100
 
 diff:
 	pipenv run npx cdk diff --app cdk/app.py || true

diff --git a/Pipfile b/Pipfile
@@ -24,6 +24,8 @@ boto3 = "==1.16.52"
 xarray = "== 0.17.0"
 zarr = "==2.7.0"
 pangeo-forge = {git = "https://github.com/pangeo-forge/pangeo-forge.git" }
+pre-commit = "==2.10.0"
+pre-commit-hooks = "==3.4.0"
 
 [requires]
 python_version = "3.8"

diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/cdk/bakery_stack.py b/cdk/bakery_stack.py
@@ -13,9 +13,7 @@
 
 
 class BakeryStack(core.Stack):
-    def __init__(
-        self, scope: core.Construct, construct_id: str, identifier: str, **kwargs
-    ) -> None:
+    def __init__(self, scope: core.Construct, construct_id: str, identifier: str, **kwargs) -> None:
         super().__init__(scope, construct_id, **kwargs)
         bucket = aws_s3.Bucket(
             self,
@@ -48,12 +46,8 @@ def __init__(
             id=f"security-group-{identifier}",
             vpc=vpc,
         )
-        security_group.add_ingress_rule(
-            aws_ec2.Peer.any_ipv4(), aws_ec2.Port.tcp_range(8786, 8787)
-        )
-        security_group.add_ingress_rule(
-            aws_ec2.Peer.any_ipv6(), aws_ec2.Port.tcp_range(8786, 8787)
-        )
+        security_group.add_ingress_rule(aws_ec2.Peer.any_ipv4(), aws_ec2.Port.tcp_range(8786, 8787))
+        security_group.add_ingress_rule(aws_ec2.Peer.any_ipv6(), aws_ec2.Port.tcp_range(8786, 8787))
         security_group.add_ingress_rule(security_group, aws_ec2.Port.all_tcp())
         cluster = aws_ecs.Cluster(
             self,
@@ -76,9 +70,7 @@ def __init__(
         )
         ecs_task_role.add_to_policy(
             aws_iam.PolicyStatement(
-                resources=[
-                    f"arn:aws:logs:{self.region}:{self.account}:log-group:dask-ecs*"
-                ],
+                resources=[f"arn:aws:logs:{self.region}:{self.account}:log-group:dask-ecs*"],
                 actions=[
                     "logs:GetLogEvents",
                 ],
@@ -121,9 +113,7 @@ def __init__(
             ),
             port_mappings=[aws_ecs.PortMapping(container_port=8080, host_port=8080)],
             logging=aws_ecs.LogDriver.aws_logs(stream_prefix="ecs-agent"),
-            environment={
-                "PREFECT__CLOUD__AGENT__LABELS": os.environ["PREFECT_AGENT_LABELS"]
-            },
+            environment={"PREFECT__CLOUD__AGENT__LABELS": os.environ["PREFECT_AGENT_LABELS"]},
             secrets={"PREFECT__CLOUD__AGENT__AUTH_TOKEN": runner_token_secret},
             command=[
                 "--cluster",
@@ -133,17 +123,15 @@ def __init__(
             ],
         )
 
-        prefect_ecs_agent_service = (
-            aws_ecs_patterns.ApplicationLoadBalancedFargateService(
-                self,
-                id=f"prefect-ecs-agent-service-{identifier}",
-                assign_public_ip=True,
-                platform_version=aws_ecs.FargatePlatformVersion.LATEST,
-                desired_count=1,
-                task_definition=prefect_ecs_agent_task_definition,
-                cluster=cluster,
-                propagate_tags=aws_ecs.PropagatedTagSource.SERVICE,
-            )
+        prefect_ecs_agent_service = aws_ecs_patterns.ApplicationLoadBalancedFargateService(
+            self,
+            id=f"prefect-ecs-agent-service-{identifier}",
+            assign_public_ip=True,
+            platform_version=aws_ecs.FargatePlatformVersion.LATEST,
+            desired_count=1,
+            task_definition=prefect_ecs_agent_task_definition,
+            cluster=cluster,
+            propagate_tags=aws_ecs.PropagatedTagSource.SERVICE,
         )
 
         prefect_ecs_agent_service.target_group.configure_health_check(

diff --git a/example.env b/example.env
@@ -7,4 +7,3 @@ PREFECT__CLOUD__AUTH_TOKEN="my-prefect-tenant-token"
 PREFECT_PROJECT="pangeo-forge-aws-bakery"
 PREFECT_AGENT_LABELS='["dask_test"]'
 PYTHONPATH=.
-
diff --git a/flow_test/transform_flow.py b/flow_test/transform_flow.py
@@ -75,17 +75,13 @@ def source_url(day: str) -> str:
 ) as flow:
     days = Parameter(
         "days",
-        default=pd.date_range("1981-09-01", "1981-09-10", freq="D")
-        .strftime("%Y-%m-%d")
-        .tolist(),
+        default=pd.date_range("1981-09-01", "1981-09-10", freq="D").strftime("%Y-%m-%d").tolist(),
     )
     sources = source_url.map(days)
     zarr_output = "dask_transform_flow.zarr"
     nc_sources = download.map(
         sources,
-        cache_location=unmapped(
-            f"s3://{outputs['cache_bucket_name_output']}/cache/{zarr_output}"
-        ),
+        cache_location=unmapped(f"s3://{outputs['cache_bucket_name_output']}/cache/{zarr_output}"),
     )
     chunked = chunk(nc_sources, size=5)
     target = f"s3://{outputs['cache_bucket_name_output']}/target/{zarr_output}"

diff --git a/images/worker/Dockerfile b/images/worker/Dockerfile
@@ -1,5 +1,5 @@
 FROM python:3.8-buster
-RUN pip install dask-cloudprovider[aws]==2021.3.1 
+RUN pip install dask-cloudprovider[aws]==2021.3.1
 RUN pip install prefect[aws]==0.14.13
 RUN pip install xarray
 RUN pip install zarr