Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add docker containers to training #1071

Merged
merged 11 commits into from
Dec 18, 2023
Merged

add docker containers to training #1071

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5dbcb53
add docker containers to training
andrewpeng02 Dec 6, 2023
43b9b36
remove nginx
andrewpeng02 Dec 7, 2023
f8d72bb
update dockerfiles
andrewpeng02 Dec 7, 2023
da26df3
update dockerfiles
andrewpeng02 Dec 9, 2023
52e7f62
:art: Auto-generated directory tree for repository in Architecture.md
andrewpeng02 Dec 9, 2023
771743b
update readme
andrewpeng02 Dec 9, 2023
585b4a3
add env argument
andrewpeng02 Dec 11, 2023
96c94c9
fix dependencies
andrewpeng02 Dec 11, 2023
8dc94e5
change base image
andrewpeng02 Dec 13, 2023
8de1675
update dev dep
andrewpeng02 Dec 18, 2023
2fbf4df
:art: Auto-generated directory tree for repository in Architecture.md
andrewpeng02 Dec 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
282 changes: 141 additions & 141 deletions .github/Architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,220 +4,220 @@

```
📦 backend
| |- 📂 ml:
| | |- 📜 __init__.py
| | |- 📜 ml_trainer.py : train a classical machine learning learning model on the dataset
| | |- 📜 ml_model_parser.py
| |- 📂 dl:
| | |- 📜 dl_trainer.py : train a deep learning model on the dataset
| | |- 📜 __init__.py
| | |- 📜 dl_model.py : torch model based on user specifications from drag and drop
| | |- 📜 detection.py
| | |- 📜 dl_model_parser.py : parse the user specified pytorch model
| | |- 📜 dl_eval.py : Evaluation functions for deep learning models in Pytorch (eg: accuracy, loss, etc)
| |- 📂 common:
| | |- 📜 default_datasets.py : store logic to load in default datasets from scikit-learn
| | |- 📜 ai_drive.py
| | |- 📜 dataset.py : read in the dataset through URL or file upload
| | |- 📜 optimizer.py : what optimizer to use (ie: SGD or Adam for now)
| | |- 📜 __init__.py
| | |- 📜 loss_functions.py : loss function enum
| | |- 📜 kernel.py
| | |- 📜 constants.py : list of helpful constants
| | |- 📜 dataset.py : read in the dataset through URL or file upload
| | |- 📜 preprocessing.py
| | |- 📜 __init__.py
| | |- 📜 ai_drive.py
| | |- 📜 default_datasets.py : store logic to load in default datasets from scikit-learn
| | |- 📜 utils.py : utility functions that could be helpful
| | |- 📜 constants.py : list of helpful constants
| | |- 📜 email_notifier.py : Endpoint to send email notification of training results via API Gateway + AWS SES
| | |- 📜 loss_functions.py : loss function enum
| | |- 📜 preprocessing.py
| |- 📂 aws_helpers:
| | |- 📂 dynamo_db_utils:
| | | |- 📜 userprogress_db.py
| | | |- 📜 constants.py : list of helpful constants
| | | |- 📜 DynamoUnitTests.md
| | | |- 📜 trainspace_db.py
| | | |- 📜 constants.py : list of helpful constants
| | | |- 📜 dynamo_db_utils.py
| | | |- 📜 userprogress_db.py
| | | |- 📜 DynamoUnitTests.md
| | |- 📜 __init__.py
| |- 📂 ml:
| | |- 📜 ml_trainer.py : train a classical machine learning learning model on the dataset
| | |- 📜 ml_model_parser.py
| | |- 📜 __init__.py
| |- 📂 dl:
| | |- 📜 dl_model_parser.py : parse the user specified pytorch model
| | |- 📜 dl_model.py : torch model based on user specifications from drag and drop
| | |- 📜 __init__.py
| | |- 📜 dl_eval.py : Evaluation functions for deep learning models in Pytorch (eg: accuracy, loss, etc)
| | |- 📜 dl_trainer.py : train a deep learning model on the dataset
| | |- 📜 detection.py
| |- 📜 poetry.lock
| |- 📜 __init__.py
| |- 📜 data.csv : data csv file for use in the playground
| |- 📜 pyproject.toml
| |- 📜 middleware.py
| |- 📜 poetry.lock
| |- 📜 app.py : run the backend (entrypoint script)
| |- 📜 pyproject.toml
| |- 📜 epoch_times.csv
| |- 📜 app.py : run the backend (entrypoint script)
| |- 📜 data.csv : data csv file for use in the playground
```

## Frontend Architecture

```
📦 frontend
| |- 📂 layer_docs:
| | |- 📜 Softmax.md : Doc for Softmax layer
| | |- 📜 Linear.md : Doc for Linear layer
| | |- 📜 ReLU.md : Doc for ReLU later
| | |- 📜 softmax_equation.png : PNG file of Softmax equation
| |- 📂 public:
| | |- 📂 images:
| | | |- 📂 wiki_images:
| | | | |- 📜 tanh_plot.png
| | | | |- 📜 dropout_diagram.png
| | | | |- 📜 tanh_equation.png
| | | | |- 📜 conv2d2.gif
| | | | |- 📜 batchnorm_diagram.png
| | | | |- 📜 conv2d.gif
| | | | |- 📜 maxpool2d.gif
| | | | |- 📜 softmax_equation.png : PNG file of Softmax equation
| | | | |- 📜 sigmoid_equation.png
| | | | |- 📜 avgpool_maxpool.gif
| | | |- 📂 learn_mod_images:
| | | | |- 📜 lossExample.png
| | | | |- 📜 robotImage.jpg
| | | | |- 📜 LeakyReLUactivation.png
| | | | |- 📜 neuralnet.png
| | | | |- 📜 lossExampleTable.png
| | | | |- 📜 binarystepactivation.png
| | | | |- 📜 ReLUactivation.png
| | | | |- 📜 sigmoidfunction.png
| | | | |- 📜 tanhactivation.png
| | | | |- 📜 neuronWithEquation.png
| | | | |- 📜 neuron.png
| | | | |- 📜 lossExampleEquation.png
| | | | |- 📜 sigmoidactivation.png
| | | |- 📂 logos:
| | | | |- 📂 dlp_branding:
| | | | | |- 📜 dlp-logo.png : DLP Logo, duplicate of files in public, but essential as the frontend can't read public
| | | | | |- 📜 dlp-logo.svg : DLP Logo, duplicate of files in public, but essential as the frontend can't read public
| | | | |- 📜 dsgt-logo-light.png
| | | | |- 📜 pandas-logo.png
| | | | |- 📜 github.png
| | | | |- 📜 flask-logo.png
| | | | |- 📜 react-logo.png
| | | | |- 📜 aws-logo.png
| | | | |- 📜 pytorch-logo.png
| | | | |- 📜 dsgt-logo-white-back.png
| | | | |- 📜 dsgt-logo-dark.png
| | | | |- 📜 python-logo.png
| | | | |- 📜 google.png
| | | |- 📜 demo_video.gif : GIF tutorial of a simple classification training session
| | |- 📜 manifest.json : Default React file for choosing icon based on
| | |- 📜 index.html : Base HTML file that will be initially rendered
| | |- 📜 dlp-logo.ico : DLP Logo
| | |- 📜 robots.txt
| |- 📂 src:
| | |- 📂 backend_outputs:
| | | |- 📜 model.pt : Last model.pt output
| | | |- 📜 my_deep_learning_model.onnx : Last ONNX file output
| | | |- 📜 model.pkl
| | |- 📂 pages:
| | | |- 📂 train:
| | | | |- 📜 [train_space_id].tsx
| | | | |- 📜 index.tsx
| | | |- 📜 login.tsx
| | | |- 📜 learn.tsx
| | | |- 📜 about.tsx
| | | |- 📜 _document.tsx
| | | |- 📜 feedback.tsx
| | | |- 📜 dashboard.tsx
| | | |- 📜 LearnContent.tsx
| | | |- 📜 wiki.tsx
| | | |- 📜 _app.tsx
| | | |- 📜 forgot.tsx
| | | |- 📜 settings.tsx
| | |- 📂 common:
| | | |- 📂 utils:
| | | | |- 📜 dateFormat.ts
| | | | |- 📜 firebase.ts
| | | | |- 📜 dndHelpers.ts
| | | |- 📂 redux:
| | | | |- 📜 userLogin.ts
| | | | |- 📜 store.ts
| | | | |- 📜 train.ts
| | | | |- 📜 userLogin.ts
| | | | |- 📜 backendApi.ts
| | | | |- 📜 store.ts
| | | | |- 📜 hooks.ts
| | | |- 📂 styles:
| | | | |- 📜 Home.module.css
| | | | |- 📜 globals.css
| | | | |- 📜 Home.module.css
| | | |- 📂 components:
| | | | |- 📜 Spacer.tsx
| | | | |- 📜 Footer.tsx
| | | | |- 📜 TitleText.tsx
| | | | |- 📜 DlpTooltip.tsx
| | | | |- 📜 HtmlTooltip.tsx
| | | | |- 📜 EmailInput.tsx
| | | | |- 📜 NavBarMain.tsx
| | | | |- 📜 HtmlTooltip.tsx
| | | | |- 📜 ClientOnlyPortal.tsx
| | | |- 📂 utils:
| | | | |- 📜 firebase.ts
| | | | |- 📜 dndHelpers.ts
| | | | |- 📜 dateFormat.ts
| | | | |- 📜 EmailInput.tsx
| | | | |- 📜 DlpTooltip.tsx
| | | | |- 📜 TitleText.tsx
| | |- 📂 backend_outputs:
| | | |- 📜 my_deep_learning_model.onnx : Last ONNX file output
| | | |- 📜 model.pkl
| | | |- 📜 model.pt : Last model.pt output
| | |- 📂 features:
| | | |- 📂 Train:
| | | | |- 📂 redux:
| | | | | |- 📜 trainspaceSlice.ts
| | | | | |- 📜 trainspaceApi.ts
| | | | |- 📂 constants:
| | | | | |- 📜 trainConstants.ts
| | | | |- 📂 types:
| | | | | |- 📜 trainTypes.ts
| | | | |- 📂 redux:
| | | | | |- 📜 trainspaceSlice.ts
| | | | | |- 📜 trainspaceApi.ts
| | | | |- 📂 components:
| | | | | |- 📜 TrainspaceLayout.tsx
| | | | | |- 📜 CreateTrainspace.tsx
| | | | | |- 📜 DatasetStepLayout.tsx
| | | | |- 📂 features:
| | | | | |- 📂 Image:
| | | | | | |- 📂 redux:
| | | | | | | |- 📜 imageActions.ts
| | | | | | | |- 📜 imageApi.ts
| | | | | | |- 📂 constants:
| | | | | | | |- 📜 imageConstants.ts
| | | | | | |- 📂 types:
| | | | | | | |- 📜 imageTypes.ts
| | | | | | |- 📂 redux:
| | | | | | | |- 📜 imageApi.ts
| | | | | | | |- 📜 imageActions.ts
| | | | | | |- 📂 components:
| | | | | | | |- 📜 ImageReviewStep.tsx
| | | | | | | |- 📜 ImageFlow.tsx
| | | | | | | |- 📜 ImageDatasetStep.tsx
| | | | | | | |- 📜 ImageTrainspace.tsx
| | | | | | | |- 📜 ImageReviewStep.tsx
| | | | | | | |- 📜 ImageParametersStep.tsx
| | | | | | | |- 📜 ImageTrainspace.tsx
| | | | | | |- 📜 index.ts
| | | | | |- 📂 Tabular:
| | | | | | |- 📂 redux:
| | | | | | | |- 📜 tabularActions.ts
| | | | | | | |- 📜 tabularApi.ts
| | | | | | |- 📂 constants:
| | | | | | | |- 📜 tabularConstants.ts
| | | | | | |- 📂 types:
| | | | | | | |- 📜 tabularTypes.ts
| | | | | | |- 📂 redux:
| | | | | | | |- 📜 tabularActions.ts
| | | | | | | |- 📜 tabularApi.ts
| | | | | | |- 📂 components:
| | | | | | | |- 📜 TabularParametersStep.tsx
| | | | | | | |- 📜 TabularDatasetStep.tsx
| | | | | | | |- 📜 TabularTrainspace.tsx
| | | | | | | |- 📜 TabularFlow.tsx
| | | | | | | |- 📜 TabularReviewStep.tsx
| | | | | | | |- 📜 TabularDatasetStep.tsx
| | | | | | | |- 📜 TabularParametersStep.tsx
| | | | | | | |- 📜 TabularFlow.tsx
| | | | | | |- 📜 index.ts
| | | | |- 📂 components:
| | | | | |- 📜 CreateTrainspace.tsx
| | | | | |- 📜 DatasetStepLayout.tsx
| | | | | |- 📜 TrainspaceLayout.tsx
| | | |- 📂 Feedback:
| | | | |- 📂 redux:
| | | | | |- 📜 feedbackApi.ts
| | | |- 📂 Dashboard:
| | | | |- 📂 redux:
| | | | | |- 📜 dashboardApi.ts
| | | | |- 📂 components:
| | | | | |- 📜 TrainDataGrid.tsx
| | | | | |- 📜 TrainBarChart.tsx
| | | | | |- 📜 TrainDoughnutChart.tsx
| | | |- 📂 LearnMod:
| | | | |- 📜 MCQuestion.tsx
| | | | |- 📜 ImageComponent.tsx
| | | | |- 📜 ModulesSideBar.tsx
| | | | |- 📜 Exercise.tsx
| | | | |- 📜 ModulesSideBar.tsx
| | | | |- 📜 FRQuestion.tsx
| | | | |- 📜 LearningModulesContent.tsx
| | | | |- 📜 ClassCard.tsx
| | | | |- 📜 ImageComponent.tsx
| | | | |- 📜 LearningModulesContent.tsx
| | | |- 📂 OpenAi:
| | | | |- 📜 openAiUtils.ts
| | |- 📂 pages:
| | | |- 📂 train:
| | | | |- 📜 index.tsx
| | | | |- 📜 [train_space_id].tsx
| | | |- 📜 login.tsx
| | | |- 📜 feedback.tsx
| | | |- 📜 learn.tsx
| | | |- 📜 dashboard.tsx
| | | |- 📜 about.tsx
| | | |- 📜 _document.tsx
| | | |- 📜 _app.tsx
| | | |- 📜 settings.tsx
| | | |- 📜 forgot.tsx
| | | |- 📜 LearnContent.tsx
| | | |- 📜 wiki.tsx
| | |- 📜 iris.csv : Sample CSV data
| | |- 📜 GlobalStyle.ts
| | | |- 📂 Dashboard:
| | | | |- 📂 redux:
| | | | | |- 📜 dashboardApi.ts
| | | | |- 📂 components:
| | | | | |- 📜 TrainDoughnutChart.tsx
| | | | | |- 📜 TrainBarChart.tsx
| | | | | |- 📜 TrainDataGrid.tsx
| | |- 📜 constants.ts
| | |- 📜 GlobalStyle.ts
| | |- 📜 next-env.d.ts
| |- 📜 next.config.js
| | |- 📜 iris.csv : Sample CSV data
| |- 📂 public:
| | |- 📂 images:
| | | |- 📂 learn_mod_images:
| | | | |- 📜 binarystepactivation.png
| | | | |- 📜 neuron.png
| | | | |- 📜 lossExampleEquation.png
| | | | |- 📜 ReLUactivation.png
| | | | |- 📜 neuronWithEquation.png
| | | | |- 📜 lossExampleTable.png
| | | | |- 📜 robotImage.jpg
| | | | |- 📜 lossExample.png
| | | | |- 📜 LeakyReLUactivation.png
| | | | |- 📜 tanhactivation.png
| | | | |- 📜 sigmoidactivation.png
| | | | |- 📜 neuralnet.png
| | | | |- 📜 sigmoidfunction.png
| | | |- 📂 wiki_images:
| | | | |- 📜 tanh_equation.png
| | | | |- 📜 tanh_plot.png
| | | | |- 📜 conv2d.gif
| | | | |- 📜 conv2d2.gif
| | | | |- 📜 dropout_diagram.png
| | | | |- 📜 batchnorm_diagram.png
| | | | |- 📜 maxpool2d.gif
| | | | |- 📜 avgpool_maxpool.gif
| | | | |- 📜 softmax_equation.png : PNG file of Softmax equation
| | | | |- 📜 sigmoid_equation.png
| | | |- 📂 logos:
| | | | |- 📂 dlp_branding:
| | | | | |- 📜 dlp-logo.svg : DLP Logo, duplicate of files in public, but essential as the frontend can't read public
| | | | | |- 📜 dlp-logo.png : DLP Logo, duplicate of files in public, but essential as the frontend can't read public
| | | | |- 📜 pytorch-logo.png
| | | | |- 📜 dsgt-logo-white-back.png
| | | | |- 📜 flask-logo.png
| | | | |- 📜 aws-logo.png
| | | | |- 📜 google.png
| | | | |- 📜 python-logo.png
| | | | |- 📜 dsgt-logo-dark.png
| | | | |- 📜 pandas-logo.png
| | | | |- 📜 react-logo.png
| | | | |- 📜 github.png
| | | | |- 📜 dsgt-logo-light.png
| | | |- 📜 demo_video.gif : GIF tutorial of a simple classification training session
| | |- 📜 manifest.json : Default React file for choosing icon based on
| | |- 📜 index.html : Base HTML file that will be initially rendered
| | |- 📜 robots.txt
| | |- 📜 dlp-logo.ico : DLP Logo
| |- 📂 layer_docs:
| | |- 📜 Linear.md : Doc for Linear layer
| | |- 📜 Softmax.md : Doc for Softmax layer
| | |- 📜 ReLU.md : Doc for ReLU later
| | |- 📜 softmax_equation.png : PNG file of Softmax equation
| |- 📜 tsconfig.json
| |- 📜 package.json
| |- 📜 jest.config.js
| |- 📜 .eslintrc.json
| |- 📜 package.json
| |- 📜 tsconfig.json
| |- 📜 next.config.js
| |- 📜 .eslintignore
| |- 📜 pnpm-lock.yaml
| |- 📜 next-env.d.ts
| |- 📜 .eslintignore
```

34 changes: 34 additions & 0 deletions training/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# pull base image
FROM condaforge/miniforge-pypy3
andrewpeng02 marked this conversation as resolved.
Show resolved Hide resolved

WORKDIR /usr/src/training

# set environment variables

# install dependencies
RUN mamba create --name dlp -y

COPY environment.yml pyproject.toml ./
andrewpeng02 marked this conversation as resolved.
Show resolved Hide resolved
RUN mamba run --live-stream -n dlp mamba env update --file environment.yml --prune
RUN mamba run --live-stream -n dlp poetry install
andrewpeng02 marked this conversation as resolved.
Show resolved Hide resolved

# create directory for the app user
RUN mkdir -p /home/app

# create the app user
RUN addgroup --system app && adduser --system --group app
dwu359 marked this conversation as resolved.
Show resolved Hide resolved

# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web
RUN mkdir $APP_HOME
WORKDIR $APP_HOME

# copy project
COPY . $APP_HOME

# chown all the files to the app user
RUN chown -R app:app $APP_HOME

# change to the app user
USER app
Loading
Loading