Skip to content

COSMOS Ingestion on ML4AI Lab Servers

Jump to bottom
Clayton T. Morrison edited this page Sep 17, 2021 · 7 revisions

COSMOS Ingestion Pipeline Setup and Usage

Last updated: 09/16/2021

This page documents how to setup and run the COSMOS ingestion pipeline on the various ML4AI servers. Currently the only supported server is kraken and the pipeline is failing with a known issue on carp (documented below).

Initial server setup

The following is a list of required software (paired with setup guides) that will install all pre-requisite software on a new ML4AI lab server needed to run the COSMOS Ingestion Pipeline. This is only required when a new (GPU-enabled) server is being setup.

NOTE: these instructions are intended for a server running the Ubuntu 20.04 operating system.

  1. Install Nvidia graphics drivers
  2. Install Nvidia CUDA drivers
  3. Install Nvidia CuDNN drivers
  4. Verify the Nvidia CUDA/CuDNN by downloading and running the MNIST sample located here:
  5. Install the docker-ci runtime environment
  6. Install Nvidia-docker (sometimes called nvidia-docker2 or nvidia-container-toolkit)
  7. Install docker-compose (ensure this is an up-to-date version -- 1.29.2 works well)
  8. Restart the Docker daemon
  9. Verify that the installed GPUs are reachable in docker using the following command:

Ingestion pipeline setup

  • Ensure your user account is in the ai and docker user groups
  • Clone the Cosmos repository to a location in your home directory (for this working example we will use /home/my_username/Cosmos to refer to that location)
  • Create a data directory structure to store COSMOS input, temp and output files.
    • The root of my data directory structure will be /home/my_username/COSMOS-data/.
    • Add the following sub-directories:
      • /home/my_username/COSMOS-data/input_files/
      • /home/my_username/COSMOS-data/output_files/
      • /home/my_username/COSMOS-data/tmp_files/
    • Change the file permissions on the root dir and all child dirs to allow full access with the command:
      • chmod -R 777 /home/my_username/COSMOS-data/
      • This is necessary for COSMOS to read/write data in these dirs (execution access (x) is also needed for some reason)
  • Create the file .env within /home/my_username/Cosmos/deployment with the content that follows. NOTE: you will need to replace the <absolute_path_to_COSMOS_data> for INPUT_DIR, TMP_DIR, and OUTPUT_DIR with the appropriate absolute path, which in our working example here is /home/my_username/COSMOS-data.
BASE_IMAGE=uwcosmos/cosmos-base-cu111:latest
DETECT_IMAGE=uwcosmos/cosmos-ingestion-cu111:latest
WORKER_IMAGE=uwcosmos/cosmos-ingestion-cu111:latest
RETRIEVAL_IMAGE=uwcosmos/cosmos-retrieval:latest
EXTRACTION_IMAGE=ankurgos/cosmos-extraction:latest
VISUALIZER_IMAGE=uwcosmos/visualizer_kb:latest
LINKING_IMAGE=uwcosmos/cosmos-linking:latest
UPLOAD_IMAGE=uwcosmos/cosmos-api:latest
API_IMAGE=uwcosmos/cosmos-api:latest

SCHEDULER_ADDRESS=scheduler:8786
ELASTIC_ADDRESS=es01:9200
NUM_PROCESSES=$WORKER_PROCS

WORKER_PROCS=4
DETECT_PROCS=1

# Default to GPU
DEVICE=cuda
RERANKING_DEVICE=cuda

# Uncomment to use CPUs
#DEVICE=cpu
#RERANKING_DEVICE=cpu

# Env vars for training
TRAINING_DIR=/path/to/training_data
VALIDATION_DIR=/path/to/validation_data/
CONFIG_DIR=${PWD}/deployment/configs
WEIGHTS_DIR=/path/for/training/output

# Env vars for primary pipeline
INPUT_DIR=<absolute_path_to_COSMOS_data>/input_files
TMP_DIR=<absolute_path_to_COSMOS_data>/tmp_files
OUTPUT_DIR=<absolute_path_to_COSMOS_data>/output_files
ELASTIC_DATA_PATH=/path/for/elasticsearch/

Transferring data to/from ML4AI Lab servers

Running the ingestion pipeline

Best practices

Known issues

Clone this wiki locally