Skip to content

Commit

Permalink
Merge branch 'main' into release
Browse files Browse the repository at this point in the history
  • Loading branch information
@deepansh96
deepansh96 committed Apr 1, 2023
2 parents 59f429f + 1aa6e91 commit 3376b4f
Show file tree
Hide file tree
Showing 3 changed files with 221 additions and 28 deletions.
2 changes: 1 addition & 1 deletion .env.example
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
MONGO_AUTH_CREDENTIALS=""
MONGO_AUTH_CREDENTIALS="mongodb://127.0.0.1/quiz"
168 changes: 141 additions & 27 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,59 +9,173 @@ The backend for a generic mobile-friendly quiz engine created using FastAPI and

## Installation

### Local DB Setup

- The following steps are for a Mac.
- To run the backend locally, you would need to setup a local instance of mongodb. The offical instructions [here](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-os-x/#run-mongodb-community-edition) are good and simple enough to follow. Those steps are also listed down below.

- Install the Xcode command-line tools
```bash
xcode-select --install
```
- Make sure you have homebrew installed. If not, go [here](https://brew.sh/#install)
- Download the official Homebrew formula for MongoDB
```bash
brew tap mongodb/brew
```
- Update Homebrew and all existing formulae
```bash
brew update
```
- Install MongoDB
```bash
brew install mongodb-community@6.0
```
- Make sure mongosh (your mongo shell) is added to the PATH. To test it out, type `mongosh` in your terminal and press enter. It should NOT give any error.

### Virtual Environment Setup

- Create a virtual environment (make sure that `virtualenv` is installed on your system):
```bash
virtualenv venv
```

- Activate the environment:
```bash
source venv/bin/activate
```

- Install the dependencies:
```bash
pip install -r app/requirements.txt
```

- Install `pre-commit`:
```
pip install pre-commit
```

- Set up `pre-commit`:
```
pre-commit install
```

- Copy `.env.example` to `.env` and set all the environment variables as mentioned in `docs/ENV.md`. No need to change anything if you're planning to connect to a local DB. If you're planning to connect your local server to a staging and prod DB, only then you need to change. PLEASE DO NOT CONNECT YOUR LOCAL INSTANCE TO STAGING/PROD DB.

## Running locally

Simply run:

```bash
virtualenv venv
chmod +x startServer.sh
./startServer.sh
```

- Activate the environment:

You should see a message like:
```bash
source venv/bin/activate
Starting the mongod process
Service `mongodb-community` already started, use `brew services restart mongodb-community` to restart.
Starting the server now
INFO: Will watch for changes in these directories: ['/Users/deepansh/Documents/Work/repos/quiz-backend/app']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [38899] using statreload
INFO: Started server process [38901]
INFO: Waiting for application startup.
INFO: Application startup complete.
```
- Install the dependencies:
Use `http://127.0.0.1:8000` as the base URL of the endpoints and navigate to `http://127.0.0.1:8000/docs` to see the auto-generated docs! :dancer:
### How to pull the latest data from Prod / Staging DB to your local DB before you start working on a feature?
You can pass arguments to the `startServer` script.
- `--freshSync` : Passing this argument means you're telling the script to take a fresh sync from the cloud DB to your local DB. By default, this is false.
- `--source` : When `--freshSync` is specified, a source is also needed. Whether you need to sync from prod db or staging db. Please specify the full [mongo URI](https://www.mongodb.com/docs/manual/reference/connection-string/) of the DB you want to take a sync from. Note: Currently it might take 10-15 minutes for the sync process to be done. We're working on improving this.
Example:
```bash
pip install -r app/requirements.txt
./startServer.sh --freshSync --source mongodb+srv://quiz:<YOUR-PASSWORD>@quiz-staging-m10.uocfg.mongodb.net/quiz
```
- Install `pre-commit`
You should see a message like:
```bash
Starting the mongod process
Service `mongodb-community` already started, use `brew services restart mongodb-community` to restart.
Fresh sync is true -- Going to remove current db and take a fresh sync

```
pip install pre-commit
```

- Set up `pre-commit`
Removing the existing data in the local database
Current Mongosh Log ID: 642317a1139325e074af0309
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.8.0
Using MongoDB: 6.0.5
Using Mongosh: 1.8.0

```
pre-commit install
```
For mongosh info see: https://docs.mongodb.com/mongodb-shell/

- Copy `.env.example` to `.env` and set all the environment variables as mentioned in `docs/ENV.md`.
------
The server generated these startup warnings when booting
2023-03-27T15:17:54.237+05:30: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2023-03-27T15:17:54.237+05:30: Soft rlimits for open file descriptors too low
------

## Running locally
------
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
Simply run:
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
```
cd app; uvicorn main:app --reload
```
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
------
You should see a message like:
{ ok: 1, dropped: 'quiz' }
quiz DB dropped
```
Taking a fresh sync from staging DB:
Downlading the data from staging DB
2023-03-28T22:06:51.043+0530 WARNING: On some systems, a password provided directly in a connection string or using --uri may be visible to system status programs such as `ps` that may be invoked by other users. Consider omitting the password to provide it via stdin, or using the --config option to specify a configuration file with the password.
2023-03-28T22:06:52.228+0530 writing quiz.quizzes to dump/quiz/quizzes.bson
2023-03-28T22:06:52.264+0530 writing quiz.session_answers to dump/quiz/session_answers.bson
2023-03-28T22:06:52.300+0530 writing quiz.sessions to dump/quiz/sessions.bson
2023-03-28T22:06:52.377+0530 writing quiz.questions to dump/quiz/questions.bson
2023-03-28T22:06:54.063+0530 [###.....................] quiz.quizzes 101/611 (16.5%)
2023-03-28T22:06:54.064+0530 [........................] quiz.session_answers 101/7868281 (0.0%)
2023-03-28T22:06:54.064+0530 [........................] quiz.sessions 101/121840 (0.1%)
2023-03-28T22:06:54.064+0530 [........................] quiz.questions 101/18467 (0.5%)
Data downloaded from staging DB, restoring the data in local DB
2023-03-28T22:40:05.949+0530 using default 'dump' directory
2023-03-28T22:40:05.951+0530 preparing collections to restore from
2023-03-28T22:40:06.034+0530 reading metadata for quiz.organization from dump/quiz/organization.metadata.json
2023-03-28T22:40:06.038+0530 reading metadata for quiz.questions from dump/quiz/questions.metadata.json
2023-03-28T22:40:06.038+0530 reading metadata for quiz.quizzes from dump/quiz/quizzes.metadata.json
2023-03-28T22:40:06.038+0530 reading metadata for quiz.session_answers from dump/quiz/session_answers.metadata.json
2023-03-28T22:40:06.039+0530 reading metadata for quiz.sessions from dump/quiz/sessions.metadata.json
Removing the downloaded dump folder
Starting the server now
INFO: Will watch for changes in these directories: ['/Users/deepansh/Documents/Work/repos/quiz-backend/app']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [98098] using watchgod
INFO: Started server process [98100]
INFO: Started reloader process [37464] using statreload
INFO: Started server process [37466]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 127.0.0.1:58550 - "GET /docs HTTP/1.1" 200 OK
INFO: 127.0.0.1:58550 - "GET /openapi.json HTTP/1.1" 200 OK
```
Use `http://127.0.0.1:8000` as the base URL of the endpoints and navigate to `http://127.0.0.1:8000/docs` to see the auto-generated docs! :dancer:
#### What's happening above?
- mongo service is started using homebrew if not already started
- existing db is cleared
- a data dump from staging/prod db is taken
- the downloaded data is restored to the local db
- the downloaded dump is deleted
- virtual environment is started up and uvicorn app is started

## Deployment

Expand Down
79 changes: 79 additions & 0 deletions startServer.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#!/bin/bash

# Default values for arguments
freshSync=false
source=""

# Parse command line arguments
# Two arguments are expected: --freshSync and --source
# --freshSync is a boolean argument, which when specified true, will start the server with a fresh sync from the source DB
# --source is a string argument, the mongo DB uri, which tells the script which cloud DB to use for the sync
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
-f|--freshSync)
freshSync=true
shift
;;
-s|--source)
if [[ -z "$2" ]] || [[ "$2" =~ ^-.* ]]; then
echo "Error: --source requires a non-empty argument"
exit 1
fi
source="$2"
shift
shift
;;
*)
echo "Invalid argument: $1"
exit 1
;;
esac
done

# Check if freshSync is true and source is empty
if $freshSync && [[ -z $source ]]; then
echo "Error: --freshSync requires --source to be specified"
exit 1
fi

# Check if source is specified and freshSync is false
if [[ -n $source ]] && ! $freshSync; then
echo "Error: --source requires --freshSync to be specified"
exit 1
fi

# start the mongod process
echo "Starting the mongod process"
brew services start mongodb-community@6.0

# Check the value of freshSync and start the sync process if true
if [ "$freshSync" = true ]; then
echo -e "Fresh sync is true -- Going to remove current db and take a fresh sync \n \n"
echo "Removing the existing data in the local database"
mongosh --eval "db.getSiblingDB('quiz').dropDatabase()"
echo "quiz DB dropped"
echo -e "\n \n"

echo -e "Taking a fresh sync from the specified DB: \n \n"
echo "Downlading the data from DB"
mongodump --uri $source
if [ $? -eq 1 ]; then
echo "Error while downloading the data from the cloud DB. Please check your mongo DB URI and try again"
exit 1
fi
echo "Data downloaded from cloud DB, restoring the data in local DB"
mongorestore --uri mongodb://127.0.0.1
echo "Data restored in local DB!"

echo -e "\n \n"
echo "Removing the downloaded dump folder"
rm -rf dump
fi

echo "Starting the server now"
source venv/bin/activate
cd app/
uvicorn main:app --reload

0 comments on commit 3376b4f

Please sign in to comment.