UPDATE 12/12/2023: killed deployed instance. Run locally to use this program.

HERRKOS - [Con]struction [Saf]ety

Using the Herrkos API - Documentation

This API allows you to detect Personal Protective Equipment (PPE) in photos of construction workers. It provides information about the identified objects, their bounding boxes, and confidence scores.

Endpoint

The API endpoint for processing images is:

POST https://api.herrkos.com/process_image

Request

To use this API, you need to make a POST request with a JSON body containing the URL of the image you want to analyze.

Request Body:

{
"image_url" : "https://images.unsplash.com/photo-1489514354504-1653aa90e34e?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w0ODI0MjV8MHwxfHJhbmRvbXx8fHx8fHx8fDE2OTA5MzY1ODB8&ixlib=rb-4.0.3&q=80&w=1080"
}

Example Request

curl -X POST -H "Content-Type: application/json" -d '{"image_url": "https://images.unsplash.com/photo-1489514354504-1653aa90e34e?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3w0ODI0MjV8MHwxfHJhbmRvbXx8fHx8fHx8fDE2OTA5MzY1ODB8&ixlib=rb-4.0.3&q=80&w=1080"}' https://api.herrkos.com/process_image

Response Upon successful processing of your request, the API will respond with a JSON object containing information about the detected objects in the image.

Response Body:

{
    "predictions": [
        {
            "box": [
                538.1019897460938,
                62.43950271606445,
                607.2695922851562,
                149.63783264160156
            ],
            "category": "Hardhat",
            "score": 0.8262612819671631
        }
    ]
}

Response Explanation "predictions": An array of detected objects in the image. "box": The coordinates of the bounding box around the detected object. "category": The name of the detected object (e.g., "Hard Hat" or "Safety Vest"). "score": A confidence score indicating the likelihood that the object is correctly identified (values range from 0 to 1, where 1 indicates high confidence).

Error Handling In case of an error, the API will respond with an appropriate HTTP status code and an error message in JSON format.

Example Error Response:

    {"error": "Image URL not provided"}

HERRKOS

Current State:
This is a construction safety tooling web API aimed at remote monitoring for your construction site.
It is currently in its infancy. As of now, the api endpoint /image_analysis GET's a random image from unsplash using the search query "construction worker." Using the image meta data, the coordinates are stored and used to reverse geo-code a location. Then, it provides the day's weather report. Finally, a Yolov5n object detection model is run to detect PPE in the image, or lack there-of. All of this data is returned as a JSON object to the frontend client.

Future State:
Query: What if we could monitor construction sites with fixed cameras and drones? What if the devices ran object detection models to detect safety complianc or for quality control/assurance checks? What if the reports could be autogenerated from this information, using historical data to send stakeholders critical information about their construction project? The idea for [Con][Saf] came from my desire to integrate bleeding-edge technology with civil engineering and construction. Construction productivity and efficiency are not growing exponentially as we see in other fields - this just means we need to take a look at the industry with fresh perspectives. There's plenty of technology here, right now, and even more coming - we just need to build smart.

HERRKOS API - Image generation with weather and object detection Demo - Screenshots

The above image shows the output from the webapp.

The above image shows the randomly generated stock photo that was analyzed with the object detection model

The above image shows the location for the returned coordinates which matches the value for the "location" key

The above image shows the weather for the given location which matches the values for "apparent_temperature_max" and apparent_temperature_min

The Future of Herrkos

[1.0.0 - Major Release]' Real time site monitoring via camera on-site

• Inputs
  • constant live video from camera with wifi service to use IP for weather information
• Outputs
  • safety object detection
  • real time weather monitoring

[0.1.0 - MVP - alpha] static site to provide insights based on user inputs

• Inputs

  • Image URL
  • User inputted location

• Outputs

  • object detection
  • weather information

Tech Stack

Frontend

• coming soon

Backend and significant pythong packages

• Flask
• yolov5
• gunicorn as production webserver on Linux

Database

• Coming Soon

Deployment

• PythonAnywhere.com

API

Set up the API

shortcut to set up from the server directory
Note that we will use Python 3.10 because we will be using yolov5
$ py -3.10 -m venv .venv && source .venv/Scripts/activate && pip install -r requirements.txt

If you need to download python 10 then follow the link below and download the file. MAs you go through the Python 3.10 download wizard, ake sure you check the box that adds python to the PATH variable

download here and select this executable file:

python-3.10.11-amd64.exe 05-Apr-2023 01:09 29037240

Ensure you select the correct python interprettor
Open up the VSCode command pallete
CTRL + shift + P and type in the search bar "Select Python Interprettor" -> "Select Path" and copy the path from :

backend/              
├── api/
│   ├── .venv/       
│     ├── Scripts/   
│        ├── python.exe # copy the path to this  

Run the server in development environment from the api directory

$ flask --app main run --debug

Run the server using Docker

First, open up docker desktop on your machine and run these commands from the server directory

build the image
$ docker build -t image_consaf_api . # note the trailing "."

spin up the container
$ docker run -d --name consaf_api_container -p 80:80 image_consaf_api

Your directory should look this, so all your calls should be from the server directory and NOT the api directory myName@LAPTOP MINGW64 /myMachinePath/consaf/backend

.
├── api
│   ├── __init__.py
│   └── main.py
├── Dockerfile
└── requirements.txt

Your dockerfile should look like this:

FROM python:3.10

WORKDIR /code

COPY ./requirements.txt /code/requirements.txt

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

COPY ./api /code/api

CMD ["gunicorn", "api.main:app", "--proxy-headers", "--host", "0.0.0.0", "--port", "80"]

If you make changes to the code, you'll have to recreate the image and spin back up.

To simply start and stop the existing container,
$ docker start consaf_api_container
$ docker stop consaf_api_container

Client

Computer Vision

the object detection model uses yolov5 - model source dataset

Testing

For testing, make sure you create a new virtual environment and install the requirements_test.txt file

run pytest tests from the "backend" directory
myName@LAPTOP MINGW64 /myMachinePath/consaf/backend $ pytest tests

Debugging

API Start Up

If you get an error regarding the import of packages when trying to run your API:

• first, make sure you're using absolute paths for importing your packages

• If the errors look like this: ModuleNotFoundError: No module named 'api'
  or
  ImportError: attempted relative import with no known parent package

You will need to edit the PYTHONPATH variable. You can do this two different ways (I recommend the 1st as this is what I did and it worked for me). Make sure you have your virtualenvironment (.venv) running:

1. You'll be adding a file to the .venv directory. Navigate to the folder shown below and add a file called consaf.pth

server/
├─ .venv/
│  ├─ Lib/
│  │  ├─ site-packages/
│  │     ├─ consaf.pth

In this file, add this to it:C:\yourComputerPath\consaf\backend
Done!

2. run this command: export PYTHONPATH=$PYTHONPATH:/path/to/your/project

TO-DO - 0.0.1

FrontEnd

• Set up Home page with Login
• guest login dashboard for demos

Backend

• set up unit tests
• Set up computer vision model endpoint
• Add weather third party API
• Add Unsplash random image generator third party API
• Add reverse GEoCode from Unsplash image third party API
• Add cache strategy for location inputs when users search for weather updates
• Add rate limiting
• Set up JWT authentication
• Set Up Oauth for sign ups/login and sender for SMTP email communication but only if a user has been authenticated and has a valid JWT
• add OpenAI to report back on PPE and weather findings
• add postgreSQL db to save users along with their queried images and chatGPT responses

DevOps

• purchase domain on AWS (SSL/TLS)
• Containerize server with Docker
• deploy server using docker on AWS EC2 and db on RDS. S3 bucket to host static files

TO-DO - 0.0.2

Features

• add functionality to upload an image and have it display on the browser

FrontEnd

• TypeScript
• Sign Up with google OAuth

Backend

• use sender for SMTP email verification
• cloudflare turnstile/recaptchav3 middleware for account creation
• sign up with google

DevOps

• fully automate deployments via github actions from commits
• Load Test with Grafana K6
• Error logging with Sentry SDK

Third Party API's

• Geocode Maps for finding locations from coordinates

• Open-Meteo for weather data from coordinates

• Unsplash for random images

Testing

Navigate to the backend/api directory. From here, start up a new virtual environmnet for testing and install all the testing dependencies. Then run pytest tests