Data Lab Project

Data Lab Project provides advanced analytics and query services on various document sources like images, video streams, text documents, file system. This project is work in progress.

Features

• File system indexing - queries on file system meta-data
• Image meta-data indexing - queries on exif and geo-location meta data.
• Video meta-data indexing - queries on exif and geo-location meta data.
• Image content object recognition - queries on objects contained in images.

Microservices

• data-scanner-service - [microservice] simple service for scanning file system.
• ml-services - [microservices] simple services utilizing using machine learning.
• message-broker - [microservice]

Components

• file-system-service - [library] simple library for scanning file system.
• elasticsearch-service - [library] service for easy ElasticSearch read/write access.
• data-scanner-service - [library] service for scanning data directory and annotating data files.

Architecture

Technology stack

• Microservices - REST, Message Broker integrations, K8s, WIP
• ElasticSearch 7.15.x - main meta-data database
• Kibana 7.15.x - basic data visualizations
• RabbitMQ 3.8 - message broker
• Java 11 - microservice implementations
• Python 3.8.x - microservice implementations
• TensorFlow 2.0 / Keras - ML related tasks
• Gradle 7.2.x - build system or later
• Ubuntu 20.04 LTS - default target environment

Build, Test and Run

gradle clean installDist distZip test

Follow this user guide to run microservices locally.