Skip to content

opendilab/DI-store

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Introduction

Decision AI Store

Installation

Prerequisites
  • Linux or macOS

  • Python >= 3.6

pip install .

Quick Start

Start Etcd Server
di_store etcd_server ./conf/di_store.yaml
Start Node Tracker
di_store node_tracker ./conf/di_store.yaml
Start Storage Server
di_store storage_server ./conf/di_store.yaml
Start Storage Client
from di_store import Client
client = Client('./conf/di_store.yaml')
ref = client.put(b'Hello world.')
data = client.get(ref)
print('data:', data)
client.delete(ref)

Object prefetch

start storage server with group

di_store storage_server ./conf/di_store.yaml --group actor

or register group in client

client.register_group('actor')

specify prefetch_group when put object

ref = client.put(b'Hello world.', prefetch_group='actor')

When an object is put with prefetch_group specified, a background schedule is triggered to spread the object to all nodes corresponding to the group. The following diagram illustrates the data dissemination.

prefetch

Running DI-store on Kubernetes

k8s/README.md

Directory Structure

├── di_store: python related codes
│     ├── bin: executable files needed by "etcd_server", "node_tracker" and "storage_server" commands
│     ├── cmd: driver for "di_store" command
│     ├── common: utilities for common
│     ├── driver
│     │     └── etcd_server_driver.py: etcd_server driver
│     ├── node_tracker
│     │     └── node_tracker_client.py: node_tracker client
│     ├── storage
│     │     └── storage_client.py: storage client
│     └── tracing
│           └── tracing.py: utilities for tracing
└── go: golang related codes
      ├── fb: generated by the FlatBuffers compiler
      ├── metadata: etcd client operations
      ├── node_tracker
      │     ├── fetch.go: fetch implementation of node_tracker
      │     ├── main
      │     │     └── node_tracker.go: the main entry point for node_tracker
      │     ├── node_tracker.go
      │     └── node_tracker_client.go
      ├── pb: generated by ProtoBuffers
      ├── plasma_client: golang codes that wraps libplasma
      ├── plasma_server: golang codes that wraps executable plasma-store-server
      ├── protos: .proto and .fbs files
      ├── release: outputs of command "go build", executable files of node_tracker and storage_server
      ├── storage_server
      │     ├── fetch.go: fetch implementation of storage_server
      │     ├── main
      │     │     └── storage_server.go: the main entry point for storage_server
      │     └── storage_server.go
      ├── tracing: utilities for tracing
      └── util

License

DI-store released under the Apache 2.0 license.