From 0096c028c8027c765db7769b68a1859155f22ba2 Mon Sep 17 00:00:00 2001
From: emiglietta <58238709+emiglietta@users.noreply.github.com>
Date: Thu, 2 May 2024 14:48:59 -0700
Subject: [PATCH 1/3] Added additional EBS_VOLUME_SIZEs for when downloading
from another bucket
---
documentation/DCP-documentation/config_examples.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/documentation/DCP-documentation/config_examples.md b/documentation/DCP-documentation/config_examples.md
index 0794e5b..7b76afc 100644
--- a/documentation/DCP-documentation/config_examples.md
+++ b/documentation/DCP-documentation/config_examples.md
@@ -43,7 +43,7 @@ Our internal configurations for each pipeline are as follows:
| TASKS_PER_MACHINE | 1 | 1 | 1 | 1 | 1 | |
| MACHINE_TYPE | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | Historically we have used m4.xlarge and then m5.xlarge however very recently we have been having a hard time getting m class machines so we have switched to c class. Note that they have different memory sizes so you need to make sure MEMORY is set correctly if changing between classes. |
| MACHINE_PRICE | .20 | .20 | .20 | .20 | .20 | Will be different for different size/classes of machines. |
-| EBS_VOL_SIZE | 22 | 22 | 22 | 22 | 22 | You might need to make this larger if you set DOWNLOAD_FILES = True |
+| EBS_VOL_SIZE
(if using mounted volume)
(if downloading from external bucket) | 22
22 | 22
200 | 22
22 | 22
22 | 22
40 | Suggested size increases when downloading files from another bucket (DOWNLOAD_FILES = True) depending on the files. |
| DOWNLOAD_FILES | 'False' | 'False' | 'False' | 'False' | 'False' | |
| DOCKER_CORES | 4 | 4 | 4 | 4 | 3 | If using c class machines and large images (2k + pixels) then you might need to reduce this number. |
| CPU_SHARES | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | We never change this. |
From 730a6eecbb3af9b24651dae3a7567e0a3a7a8fa7 Mon Sep 17 00:00:00 2001
From: Erin Weisbart <54687786+ErinWeisbart@users.noreply.github.com>
Date: Fri, 3 May 2024 07:23:24 -0700
Subject: [PATCH 2/3] formatting, clarity
---
documentation/DCP-documentation/config_examples.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/documentation/DCP-documentation/config_examples.md b/documentation/DCP-documentation/config_examples.md
index 7b76afc..0529451 100644
--- a/documentation/DCP-documentation/config_examples.md
+++ b/documentation/DCP-documentation/config_examples.md
@@ -43,7 +43,8 @@ Our internal configurations for each pipeline are as follows:
| TASKS_PER_MACHINE | 1 | 1 | 1 | 1 | 1 | |
| MACHINE_TYPE | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | Historically we have used m4.xlarge and then m5.xlarge however very recently we have been having a hard time getting m class machines so we have switched to c class. Note that they have different memory sizes so you need to make sure MEMORY is set correctly if changing between classes. |
| MACHINE_PRICE | .20 | .20 | .20 | .20 | .20 | Will be different for different size/classes of machines. |
-| EBS_VOL_SIZE
(if using mounted volume)
(if downloading from external bucket) | 22
22 | 22
200 | 22
22 | 22
22 | 22
40 | Suggested size increases when downloading files from another bucket (DOWNLOAD_FILES = True) depending on the files. |
+| EBS_VOL_SIZE (if mounting S3 as a file system) | 22 | 22 | 22 | 22 | 22 | S3 is mounted as a file system when `DOWNLOAD_FILES = False`. |
+| EBS_VOL_SIZE (if downloading files) | 22 | 200 | 22 | 22 | 40 | Files are downloaded to the EBS volume when `DOWNLOAD_FILES = True`. |
| DOWNLOAD_FILES | 'False' | 'False' | 'False' | 'False' | 'False' | |
| DOCKER_CORES | 4 | 4 | 4 | 4 | 3 | If using c class machines and large images (2k + pixels) then you might need to reduce this number. |
| CPU_SHARES | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | DOCKER_CORES * 1024 | We never change this. |
From b23836afcf014c74badfa132b2f3e9b170733b0d Mon Sep 17 00:00:00 2001
From: Erin Weisbart <54687786+ErinWeisbart@users.noreply.github.com>
Date: Fri, 3 May 2024 07:28:53 -0700
Subject: [PATCH 3/3] clarity
---
documentation/DCP-documentation/config_examples.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/documentation/DCP-documentation/config_examples.md b/documentation/DCP-documentation/config_examples.md
index 0529451..671a1e7 100644
--- a/documentation/DCP-documentation/config_examples.md
+++ b/documentation/DCP-documentation/config_examples.md
@@ -43,7 +43,7 @@ Our internal configurations for each pipeline are as follows:
| TASKS_PER_MACHINE | 1 | 1 | 1 | 1 | 1 | |
| MACHINE_TYPE | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | ['c5.xlarge'] | Historically we have used m4.xlarge and then m5.xlarge however very recently we have been having a hard time getting m class machines so we have switched to c class. Note that they have different memory sizes so you need to make sure MEMORY is set correctly if changing between classes. |
| MACHINE_PRICE | .20 | .20 | .20 | .20 | .20 | Will be different for different size/classes of machines. |
-| EBS_VOL_SIZE (if mounting S3 as a file system) | 22 | 22 | 22 | 22 | 22 | S3 is mounted as a file system when `DOWNLOAD_FILES = False`. |
+| EBS_VOL_SIZE (if using S3 mounted as a file system) | 22 | 22 | 22 | 22 | 22 | Files are read directly off of S3, mounted as a file system when `DOWNLOAD_FILES = False`. |
| EBS_VOL_SIZE (if downloading files) | 22 | 200 | 22 | 22 | 40 | Files are downloaded to the EBS volume when `DOWNLOAD_FILES = True`. |
| DOWNLOAD_FILES | 'False' | 'False' | 'False' | 'False' | 'False' | |
| DOCKER_CORES | 4 | 4 | 4 | 4 | 3 | If using c class machines and large images (2k + pixels) then you might need to reduce this number. |