Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker containerisation of MVP #43

Merged
merged 15 commits into from
Mar 13, 2023
Merged

Docker containerisation of MVP #43

merged 15 commits into from
Mar 13, 2023

Conversation

shubhamlatkar
Copy link
Contributor

@shubhamlatkar shubhamlatkar commented Mar 3, 2023

  • Use Docker to automate the build for the project, using the current dependencies and infrastructure
  • Update the README.md to provide instructions on how to use the container(s)

@shubhamlatkar shubhamlatkar linked an issue Mar 3, 2023 that may be closed by this pull request
2 tasks
@AbhinavReddy-Dev
Copy link
Contributor

@shubhamlatkar Can you please add the below bind commands for running the docker with local changes?

  • Plain run
    docker run --mount type=bind,source="$(pwd)",target=/usr/src -it <image-tag-here>
  • With shell
    docker run --mount type=bind,source="$(pwd)",target=/usr/src -it <image-tag-here> sh

Copy link
Contributor

@AbhinavReddy-Dev AbhinavReddy-Dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed the changes made and they seem to be necessary. Also can we add the bind commands in the README.md? Another warning that I came across was regrading the image's platform when building the image on arm64 devices, I shall look into it and update, please update if we already have any idea of getting rid of the warning.

@shubhamlatkar
Copy link
Contributor Author

I have reviewed the changes made and they seem to be necessary. Also can we add the bind commands in the README.md? Another warning that I came across was regrading the image's platform when building the image on arm64 devices, I shall look into it and update, please update if we already have any idea of getting rid of the warning.

Hi @AbhinavReddy-Dev could you add the logs of warnings you are getting.

@AbhinavReddy-Dev
Copy link
Contributor

@shubhamlatkar sure, here's the warning for the image platform.
abhinavreddy@abhinavreddy PantryNode % docker run --mount type=bind,source="$(pwd)",target=/usr/src -it pantry-node sh WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested /app #

@shubhamlatkar
Copy link
Contributor Author

@shubhamlatkar sure, here's the warning for the image platform.
abhinavreddy@abhinavreddy PantryNode % docker run --mount type=bind,source="$(pwd)",target=/usr/src -it pantry-node sh WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested /app #

Hi @AbhinavReddy-Dev you are mounting the wrong directory target=/usr/src but our src code is in /app.
And warning is because you have not used the --platform linux/amd64 tag in docker run cmd.
As you are using ARM64 based system and linux container is linux/amd64 based so you'll need to add --platform tag
Adding it might solve this issue.

@AbhinavReddy-Dev
Copy link
Contributor

@shubhamlatkar Thank you, I was able to run it successfully using the below command.
docker run --platform linux/amd64 --mount type=bind,source="$(pwd)",target=/usr/app -it pantry-node

@shubhamlatkar
Copy link
Contributor Author

@shubhamlatkar Thank you, I was able to run it successfully using the below command.
docker run --platform linux/amd64 --mount type=bind,source="$(pwd)",target=/usr/app -it pantry-node

Actually target=/usr/app should be target=/app

@AbhinavReddy-Dev
Copy link
Contributor

@shubhamlatkar Thank you, I was able to run it successfully using the below command.
docker run --platform linux/amd64 --mount type=bind,source="$(pwd)",target=/usr/app -it pantry-node

Actually target=/usr/app should be target=/app

yes, you're right!

@sourabhk25
Copy link
Contributor

I am getting below error while executing docker run command -

(base) sourabhkulkarni@Sourabhs-MacBook-Pro PantryNode % docker run -p 3000:3000 my-app

project@0.0.0 start
node ./bin/www

/app/node_modules/send/node_modules/http-errors/index.js:261
var name = toIdentifier(statuses.message[code])
^

TypeError: Cannot read properties of undefined (reading '100')
at forEachCode (/app/node_modules/send/node_modules/http-errors/index.js:261:45)
at Array.forEach ()
at populateConstructorExports (/app/node_modules/send/node_modules/http-errors/index.js:259:9)
at Object. (/app/node_modules/send/node_modules/http-errors/index.js:31:1)
at Module._compile (node:internal/modules/cjs/loader:1191:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1245:10)
at Module.load (node:internal/modules/cjs/loader:1069:32)
at Function.Module._load (node:internal/modules/cjs/loader:904:12)
at Module.require (node:internal/modules/cjs/loader:1093:19)
at require (node:internal/modules/cjs/helpers:108:18)

@AbhinavReddy-Dev
Copy link
Contributor

AbhinavReddy-Dev commented Mar 6, 2023

I am getting below error while executing docker run command -

(base) sourabhkulkarni@Sourabhs-MacBook-Pro PantryNode % docker run -p 3000:3000 my-app

project@0.0.0 start
node ./bin/www

/app/node_modules/send/node_modules/http-errors/index.js:261 var name = toIdentifier(statuses.message[code]) ^

TypeError: Cannot read properties of undefined (reading '100') at forEachCode (/app/node_modules/send/node_modules/http-errors/index.js:261:45) at Array.forEach () at populateConstructorExports (/app/node_modules/send/node_modules/http-errors/index.js:259:9) at Object. (/app/node_modules/send/node_modules/http-errors/index.js:31:1) at Module._compile (node:internal/modules/cjs/loader:1191:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1245:10) at Module.load (node:internal/modules/cjs/loader:1069:32) at Function.Module._load (node:internal/modules/cjs/loader:904:12) at Module.require (node:internal/modules/cjs/loader:1093:19) at require (node:internal/modules/cjs/helpers:108:18)

It doesn't look like the docker error but what is the build command that you have used?

@sourabhk25
Copy link
Contributor

sourabhk25 commented Mar 6, 2023

I am getting below error while executing docker run command -
(base) sourabhkulkarni@Sourabhs-MacBook-Pro PantryNode % docker run -p 3000:3000 my-app

project@0.0.0 start
node ./bin/www

/app/node_modules/send/node_modules/http-errors/index.js:261 var name = toIdentifier(statuses.message[code]) ^
TypeError: Cannot read properties of undefined (reading '100') at forEachCode (/app/node_modules/send/node_modules/http-errors/index.js:261:45) at Array.forEach () at populateConstructorExports (/app/node_modules/send/node_modules/http-errors/index.js:259:9) at Object. (/app/node_modules/send/node_modules/http-errors/index.js:31:1) at Module._compile (node:internal/modules/cjs/loader:1191:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1245:10) at Module.load (node:internal/modules/cjs/loader:1069:32) at Function.Module._load (node:internal/modules/cjs/loader:904:12) at Module.require (node:internal/modules/cjs/loader:1093:19) at require (node:internal/modules/cjs/helpers:108:18)

It doesn't look like the docker error but what is the build command that you have used?

I used the below command to build the docker container -
docker build --platform linux/amd64 -t my-app .

PS - The application runs on my local machine if I use my MongoDB Atlas instance, but I am getting the error mentioned only while using Docker.

FYI, I am using mac with M2 chip.

@AbhinavReddy-Dev
Copy link
Contributor

AbhinavReddy-Dev commented Mar 6, 2023

Can we make a merge of this? Please review @Jooms @sourabhk25

@shubhamlatkar
Copy link
Contributor Author

@sourabhk25

I tried many things, but I am still getting the same issue. If anyone else with a Mac confirms that changes are working for them, then I will assume the issue is in my machine and approve the review as well.

I'm getting the same issue as well.

 docker-compose up
[+] Running 2/0
 - Container database  Created                                                                                                                    0.0s
 - Container node      Created                                                                                                                    0.0s
Attaching to database, node
database  | {"t":{"$date":"2023-03-12T02:47:44.126+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true}}}
database  | {"t":{"$date":"2023-03-12T02:47:44.126+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
database  | {"t":{"$date":"2023-03-12T02:47:44.127+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrationDonors"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMigrationRecipients"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","namespace":"config.tenantSplitDonors"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"cb14b8355d93"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.4","gitVersion":"44ff59461c1353638a71e710f385a566bcd2f547","openSSLVersion":"OpenSSL 3.0.2 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2204","distarch":"x86_64","target_arch":"x86_64"}}}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.128+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
database  | {"t":{"$date":"2023-03-12T02:47:44.129+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3445M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.810+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":681}}
database  | {"t":{"$date":"2023-03-12T02:47:44.810+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
database  | {"t":{"$date":"2023-03-12T02:47:44.814+00:00"},"s":"W",  "c":"CONTROL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
database  | {"t":{"$date":"2023-03-12T02:47:44.814+00:00"},"s":"W",  "c":"CONTROL",  "id":22178,   "ctx":"initandlisten","msg":"/sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'","tags":["startupWarnings"]}
database  | {"t":{"$date":"2023-03-12T02:47:44.814+00:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}
database  | {"t":{"$date":"2023-03-12T02:47:44.816+00:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":17,"maxWireVersion":17},"outgoing":{"minWireVersion":17,"maxWireVersion":17},"isInternalClient":true}}}
database  | {"t":{"$date":"2023-03-12T02:47:44.816+00:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"6.0","context":"startup"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.816+00:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
database  | {"t":{"$date":"2023-03-12T02:47:44.817+00:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
database  | {"t":{"$date":"2023-03-12T02:47:44.818+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.820+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.820+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
database  | {"t":{"$date":"2023-03-12T02:47:44.821+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.821+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
database  | {"t":{"$date":"2023-03-12T02:47:44.821+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
node      |
node      | > project@0.0.0 start
node      | > node ./bin/www
node      |
node      | /app/node_modules/send/node_modules/http-errors/index.js:261
node      |     var name = toIdentifier(statuses.message[code])
node      |                                             ^
node      |
node      | TypeError: Cannot read properties of undefined (reading '100')
node      |     at forEachCode (/app/node_modules/send/node_modules/http-errors/index.js:261:45)
node      |     at Array.forEach (<anonymous>)
node      |     at populateConstructorExports (/app/node_modules/send/node_modules/http-errors/index.js:259:9)
node      |     at Object.<anonymous> (/app/node_modules/send/node_modules/http-errors/index.js:31:1)
node      |     at Module._compile (node:internal/modules/cjs/loader:1275:14)
node      |     at Module._extensions..js (node:internal/modules/cjs/loader:1329:10)
node      |     at Module.load (node:internal/modules/cjs/loader:1133:32)
node      |     at Module._load (node:internal/modules/cjs/loader:972:12)
node      |     at Module.require (node:internal/modules/cjs/loader:1157:19)
node      |     at require (node:internal/modules/helpers:119:18)
node      |
node      | Node.js v19.7.0
node exited with code 1

Hi @sourabhk25, @Jooms there seems to be an issue with http-errors package for ARM64 systems.
image
Any suggestion on how can we resolve it?

@Jooms
Copy link
Contributor

Jooms commented Mar 13, 2023

there seems to be an issue with http-errors package for ARM64 systems

How did you determine this?

@AbhinavReddy-Dev
Copy link
Contributor

Do we need to worry about containerizing mongoDB? We are anyways migrating it to PostgreSQL, some coordination with backend team to provide with code that doesn't have mongo references and only PostgreSQL references should be enough to try containerizing this locally and then a merge into main. Please let me know if I have overlooked any issue in this approach.

@shubhamlatkar
Copy link
Contributor Author

there seems to be an issue with http-errors package for ARM64 systems

How did you determine this?

The issue is in node_modules/http-errors/index.js
And the container is working fine on AMD64 systems

@Jooms
Copy link
Contributor

Jooms commented Mar 13, 2023

The issue is in node_modules/http-errors/index.js
And the container is working fine on AMD64 systems

I'm on an AMD 64 system... So, how did you determine this?

Processor AMD Ryzen 9 5900

@shubhamlatkar
Copy link
Contributor Author

Do we need to worry about containerizing mongoDB? We are anyways migrating it to PostgreSQL, some coordination with backend team to provide with code that doesn't have mongo references and only PostgreSQL references should be enough to try containerizing this locally and then a merge into main. Please let me know if I have overlooked any issue in this approach.

MongoDB container was added to avoid the need for setting up a local database. However, we will need to remove it once the project is migrated to PostgreSQL. For now, we can skip setting up the database container. But I think there are no issues with the MongoDB container at this time.

@shubhamlatkar
Copy link
Contributor Author

shubhamlatkar commented Mar 13, 2023

The issue is in node_modules/http-errors/index.js
And the container is working fine on AMD64 systems

I'm on an AMD 64 system... So, how did you determine this?

Processor AMD Ryzen 9 5900

It's strange because I ran the program on a cloud platform that randomly assigns x86/AMD64 systems, and it worked without any issues.
You can try upgrading http-errors pkg to latest version.

@Jooms
Copy link
Contributor

Jooms commented Mar 13, 2023

Figured it out

When the Dockerfile is run on a repo that has already been built locally, there's a node_modules folder that is copied over as part of COPY . .

Solution: Add a .dockerignore file.

Pushing a commit with the change.

@Jooms
Copy link
Contributor

Jooms commented Mar 13, 2023

Runs locally for me.

Jooms
Jooms previously approved these changes Mar 13, 2023
@Jooms Jooms mentioned this pull request Mar 13, 2023
2 tasks
@decoles
Copy link
Contributor

decoles commented Mar 13, 2023

I updated the readme to exclude the old setup instructions as they weren't useful afaik for the docker container and updated the formatting under docker setup subsection to match formatting of the rest of the document.

@shubhamlatkar
Copy link
Contributor Author

Figured it out

When the Dockerfile is run on a repo that has already been built locally, there's a node_modules folder that is copied over as part of COPY . .

Solution: Add a .dockerignore file.

Pushing a commit with the change.

Nice catch @Jooms!!!
Thanks for the fix.

@AbhinavReddy-Dev
Copy link
Contributor

Are the latest changes available in any branch to run it locally? Please quote it here.

@AbhinavReddy-Dev
Copy link
Contributor

Are the latest changes available in any branch to run it locally? Please quote it here.

NVM was able to do it.

@shubhamlatkar
Copy link
Contributor Author

Hi team can we merge the changes to the main?

@Jooms
Copy link
Contributor

Jooms commented Mar 13, 2023

Fixed merge conflict. Approving once more
@AbhinavReddy-Dev Can you approve as well since you confirmed it works for ya?

Then @shubhamlatkar Can you merge as soon as there are 2 approvals?

Copy link
Contributor

@decoles decoles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runs for me!

@shubhamlatkar shubhamlatkar merged commit 33cc130 into main Mar 13, 2023
@shubhamlatkar shubhamlatkar deleted the docker-container branch March 13, 2023 21:18
briswells pushed a commit that referenced this pull request Mar 23, 2023
- [X] Use Docker to automate the build for the project, using the
current dependencies and infrastructure
- [X] Update the `README.md` to provide instructions on how to use the
container(s)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DevOps Concerns developer operations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Containerize MVP
5 participants