/
fs_docker_ecr.Rmd
337 lines (275 loc) · 11.5 KB
/
fs_docker_ecr.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
---
title: "AWS Docker Elastic Container Registery (ECR) Update and Push"
titleshort: "AWS Docker Elastic Container Registery (ECR) Update and Push"
description: |
Update and push to Elastic Container Registry (ECR) with newly built Docker image.
Pull from Elastic Container Registry docker image.
core:
- package: scp
code: |
scp -o StrictHostKeyChecking=accept-new -i
- package: aws
code: |
aws ecr get-login
- package: docker
code: |
docker login
docker build
docker tag
docker push
docker pull
date: 2020-09-13
date_start: 2020-09-13
output:
pdf_document:
pandoc_args: '../../_output_kniti_pdf.yaml'
includes:
in_header: '../../preamble.tex'
html_document:
pandoc_args: '../../_output_kniti_html.yaml'
includes:
in_header: "../../hdga.html"
always_allow_html: true
urlcolor: blue
---
### ECR Setup
```{r global_options, include = FALSE}
try(source("../../.Rprofile"))
```
`r text_shared_preamble_one`
`r text_shared_preamble_two`
`r text_shared_preamble_thr`
#### Pull from Elastic Container Registry (ECR)
Given docker files already on ECR, in EC2, first, get password, then pull.
```{bash include=TRUE, eval = FALSE, echo = TRUE}
# log in
# copy the output from the line below and paste
aws ecr get-login --no-include-email
# this is copied from output of the command above
docker login -u AWS -p PASSWORDPASSWORDPASSWORDPASSWORD https://XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com
# pull from docker
docker pull XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com/fancondaXX
```
#### Update Elastic Container Registry (ECR)
There is a local conda file, perhaps some project repo have been updated, need to update docker file on ECR (or create new ones). Controlling EC2 can be done manually, or via [SSM](https://docs.aws.amazon.com/systems-manager/latest/userguide/prereqs-ssm-agent.html).
1. Start a EC2 Instance
2. Create a docker folder on EC2 instance (on remote)
3. scp update dockerfile in EC2 docker folder (local to remote)
4. build on remote server docker container (on remote)
5. push from EC2 updated docker to ECR (on remote)
First, after creating/starting a EC2 instance, create a docker file and scp update:
```{bash include=TRUE, eval = FALSE, echo = TRUE}
# Then a sequences of SSM calls:
# on local machine:
ssh -i "C:/Users/fan/CondaPrj/boto3aws/aws_ec2/pem/fan_wang-key-pair-us_east_nv.pem" ec2-user@54.161.29.209
# if new instance, create a docker folder under main
# on remote machine
mkdir /home/ec2-user/docker
# ssm call to remove current dockerfile
# on remote machine
rm /home/ec2-user/docker/Dockerfile
# run local scp command to copy lateste Dockerfile to EC2, local scp generated by ec2managee
# on local machine
scp -o StrictHostKeyChecking=accept-new -i C:/Users/fan/CondaPrj/boto3aws/aws_ec2/pem/fan_wang-key-pair-us_east_nv.pem C:/Users/fan/CondaPrj/boto3aws/aws_ecr/container/DockerfileInstall ec2-user@54.161.29.209:~/docker/Dockerfile
```
Second, start container service remotely, and build new container:
```{bash include=TRUE, eval = FALSE, echo = TRUE}
# start docker service on ec2
# on remote machine
sudo service docker start
# On remote machine
cd /home/ec2-user/docker
docker build -t fanconda6 --build-arg CACHE_DATE=2020-09-21-22-43-52 .
```
Third, push new container to ECR (tag, get token, login, push):
```{bash include=TRUE, eval = FALSE, echo = TRUE}
# Start Container Service
sudo service docker start
# CD into folder on remote
cd /home/ec2-user/docker
# tag docker
docker tag fancondaxxx XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com/fancondaxxx
# ECR Docker Log in
# ssm.get_authorization_token(registryIds=[boto3aws.aws_keys()['main_aws_id']])
# Decode authorization token
docker login -u AWS -p TOKENX6XXXxXXg1XXg3OX0= https://XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com
# ECR Docker Push to ECR
docker push XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com/fancondaxxx
```
#### PyFan Procedures
1. Start EC2 instance
2. Push to ECR
3. Get SSH link to EC2, SSH into EC2
4. *sudo service docker start* and *docker images* (or see pull earlier)
5. start docker image and enter to access via command line: *docker run -t -i fanconda /bin/bash*
#### More Example ECR Code and Outputs
##### Example Docker File for AWS
Note that that private git repos are pulled in. Note also that AWS keys are set up to allow for various access to AWS services.
```{bash include=TRUE, eval = FALSE, echo = TRUE}
FROM continuumio/anaconda3
VOLUME /data
# Conda update
RUN conda update conda
# https://github.com/ContinuumIO/docker-images/issues/49#issuecomment-311556456
RUN apt-get update && \
apt-get install libgl1-mesa-glx -y
# Install Conda additional packages that i use
RUN conda install -c conda-forge interpolation
RUN conda install -c conda-forge boto3
# see https://github.com/moby/moby/issues/22832, this allows for code below to run without --no-cache
ARG CACHE_DATE=2000-01-01
# Clone our private GitHub Repository: PyFan
RUN git clone https://b123451234dfc025a836927PRIVATETOKEND1239@github.com/FanWangEcon/pyfan.git /pyfan/
# Make port 80 available to the world outside this container
EXPOSE 80
# Install software
ENV PYTHONPATH /pyfan/
ENV AWS_BUCKET_NAME=BucketName
ENV AWS_ACCESS_KEY_ID=XKIXXXGSXXXBZXX43XXX
ENV AWS_SECRET_ACCESS_KEY=xxTgp9r0f4XXXXXXX1XXlG1vTy07wydxXXXXXX11
# Run
CMD ["python", "/pyfan/pyfan/graph/exa/scatterline3.py"]
```
##### Example SSM communication
```{bash include=TRUE, eval = FALSE, echo = TRUE}
# aws_keys stores keys
aws_keys_dict = aws_keys()
ssm = boto3.client('ssm',
aws_access_key_id=aws_keys_dict['aws_access_key_id'],
aws_secret_access_key=aws_keys_dict['aws_secret_access_key'],
region_name=aws_keys_dict['region'])
commands = 'rm /home/ec2-user/docker/Dockerfile'
resp = client.send_command(
DocumentName="AWS-RunShellScript", # One of AWS' preconfigured documents
Parameters={'commands': commands},
InstanceIds=[instance_id])
```
##### Outputs from Docker Build
outputs from docker build
```{bash include=TRUE, eval = FALSE, echo = TRUE}
json.py - jdump - 47 - 2020-09-22 16:04:32,459 - INFO list_command_invocation-cur_output
:[
"Sending build context to Docker daemon 3.072kB\r\r",
"Step 1/16 : FROM continuumio/anaconda3",
" ---> 472a925c4385",
"Step 2/16 : VOLUME /data",
" ---> Using cache",
" ---> cf4e6a503f00",
"Step 3/16 : RUN conda update conda",
" ---> Using cache",
" ---> 542901f01365",
"Step 4/16 : RUN apt-get update && apt-get install libgl1-mesa-glx -y",
" ---> Using cache",
" ---> 6672960aa00c",
"Step 5/16 : RUN conda install -c conda-forge interpolation",
" ---> Using cache",
" ---> efd86a4259a4",
"Step 6/16 : RUN conda install -c conda-forge boto3",
" ---> Using cache",
" ---> bd0146dac9b3",
"Step 7/16 : ARG CACHE_DATE=2000-01-01",
" ---> Using cache",
" ---> dc40688e3720",
"Step 8/16 : RUN git clone https://XXXX@github.com/FanWangEcon/pyfan.git /pyfan/",
" ---> Running in 9c0c2a444540",
"\u001b[91mCloning into '/pyfan'...",
"\u001b[0mRemoving intermediate container 9c0c2a444540",
" ---> c80480cc51a1",
"Step 9/16 : RUN git clone https://XXXX@github.com/FanWangEcon/CondaProg.git /CondaProg/",
" ---> Running in 07d9f665b760",
"\u001b[91mCloning into '/CondaProg'...",
"\u001b[0mRemoving intermediate container 07d9f665b760",
" ---> a5ac6c6e1458",
"Step 10/16 : EXPOSE 80",
" ---> Running in 1a8ef516e236",
"Removing intermediate container 1a8ef516e236",
" ---> 13ab2965e892",
"Step 11/16 : ENV PYTHONPATH /pyfan/",
" ---> Running in 2d9e4b68164b",
"Removing intermediate container 2d9e4b68164b",
" ---> 0a74e69ce1c8",
"Step 12/16 : ENV PYTHONPATH $PYTHONPATH:/CondaProg/",
" ---> Running in ba59f1273f51",
"Removing intermediate container ba59f1273f51",
" ---> 11fd9d732e2e",
"Step 13/16 : ENV AWS_BUCKET_NAME=BucketName",
" ---> Running in e7a052d3eacf",
"Removing intermediate container e7a052d3eacf",
" ---> 5e294f562838",
"Step 14/16 : ENV AWS_ACCESS_KEY_ID=XXXXX5GSDZSXXXX43XXX",
" ---> Running in 60d810a8514f",
"Removing intermediate container 60d810a8514f",
" ---> 2fa1ac4e7d3b",
"Step 15/16 : ENV AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXX",
" ---> Running in 8b34126cee5d",
"Removing intermediate container 8b34126cee5d",
" ---> 93bd8b521d61",
"Step 16/16 : CMD [\"python\", \"/CondaPrj/invoke/invoke.py\"]",
" ---> Running in dd3ed44dcca7",
"Removing intermediate container dd3ed44dcca7",
" ---> 506f92a794cd",
"Successfully built 506f92a794cd",
"Successfully tagged fanconda:latest",
"Total reclaimed space: 0B",
""
]
```
##### Output from Docker Push
```{bash include=TRUE, eval = FALSE, echo = TRUE}
json.py - jdump - 47 - 2020-09-22 16:05:32,986 - INFO list_command_invocation-cur_output
:[
"The push refers to repository [XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com/fanconda]",
"63cc929545c3: Preparing",
"d849f5d67bbb: Preparing",
"f9c77b2e4c5f: Preparing",
"7ffd6385ae0e: Preparing",
"2fc88e09d363: Preparing",
"50e089036495: Preparing",
"6637031dbcc2: Preparing",
"68d0bdfd0715: Preparing",
"d0f104dc0a1f: Preparing",
"50e089036495: Waiting",
"6637031dbcc2: Waiting",
"68d0bdfd0715: Waiting",
"d0f104dc0a1f: Waiting",
"2fc88e09d363: Layer already exists",
"7ffd6385ae0e: Layer already exists",
"50e089036495: Layer already exists",
"6637031dbcc2: Layer already exists",
"68d0bdfd0715: Layer already exists",
"d0f104dc0a1f: Layer already exists",
"d849f5d67bbb: Pushed",
"f9c77b2e4c5f: Pushed",
"63cc929545c3: Pushed",
"latest: digest: sha256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX size: 2226",
""
]
```
##### AWS ECR Instructions
Under repositories listed under ECR, click on *View push command*, which shows:
```{bash include=TRUE, eval = FALSE, echo = TRUE}
# Retrieve an authentication token and authenticate your Docker client to your registry.
# Use the AWS CLI:
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com
# Note: If you receive an error using the AWS CLI, make sure that you have the latest version of the AWS CLI and Docker installed.
# Build your Docker image using the following command. For information on building a Docker file from scratch see the instructions here . You can skip this step if your image is already built:
docker build -t fanconda .
# After the build completes, tag your image so you can push the image to this repository:
docker tag fanconda:latest XXXX7367XXXX.dkr.ecr.us-east-1.amazonaws.com/fanconda:latest
```
#### Command Line Sequence Gathered
Gathered sequence of command line operations:
```{bash include=TRUE, eval = FALSE, echo = TRUE}
ssh -i "G:/repos/CondaPrj/boto3aws/aws_ec2/pem/fan_wang-key-pair-us_east_nv.pem" ec2-user@34.229.39.138
docker run -t -i fanconda /bin/bash
scp -o StrictHostKeyChecking=accept-new -i G:/repos/CondaPrj/boto3aws/aws_ec2/pem/fan_wang-key-pair-us_east_nv.pem G:/repos/CondaPrj/boto3aws/aws_ecr/container/DockerfileConda ec2-user@34.229.39.138:~/docker/Dockerfile
sudo service docker start
cd /home/ec2-user/docker
docker build -t fanconda --build-arg CACHE_DATE=2020-12-22-10-58-57 .
docker system prune --force
cd /home/ec2-user/docker
docker tag fanconda 710673677961.dkr.ecr.us-east-1.amazonaws.com/fanconda
aws ecr get-login --no-include-email
docker login -u AWS -p XXXXX= https://710673677961.dkr.ecr.us-east-1.amazonaws.com
docker push 710673677961.dkr.ecr.us-east-1.amazonaws.com/fanconda
```