Digital Object Storage System
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.settings
bin
client
doc
project
resources-test
resources
src/doss
test/doss
.gitignore
.travis.yml
LICENSE
README.md
doss.thrift
nla-deploy.sh
pom.xml

README.md

DOSS 2

UBC Library Robot

DOSS 2 will be an object storage abstraction layer with transactional write-once semantics. Objects are uniquely and persistently identified and can never be overwritten only created and (soft) deleted.

Build Status (Huboard: Tasks, Backlog) (Javadoc) (Test coverage)

Status

Feature Goals

  • write once blob storage semantics
  • concurrent distributed random read access
  • multiple storage pools and devices (with different policies)
  • two-phase commit write transactions
  • soft deletes ("rubbish bin")
  • consistency and integrity checks
  • packs small files into containers for efficient bulk processing

Java API

See java-api.md for more examples.

Ingesting files

try (BlobStore bs = LocalBlobStore.open("/doss-devel");
    BlobTx tx = bs.begin()) {    
   
    Blob blob1 = tx.put(Paths.get("/tmp/myimage.jpg"));
    
    // if a digest is available, it's a good idea to verify it
    Blob blob2 = tx.put(Paths.get("/tmp/mytext.txt"));
    if (!blob2.digest("SHA1").equals("2aae6c35c94fcfb415dbe95f408b9ce91ee846ed")) {
        // something got corrupt so bail out (will automatically rollback)
        throw new RuntimeException("ingest failed: digest mismatch");
    }
    
    tx.commit();
    return blob1.id();
}

Random access

try (BlobStore bs = LocalBlobStore.open("/doss-devel")) {
    Blob blob = bs.get(273);
    try (Channel channel = blob.openChannel()) {
        // do something with the channel
    }
}

Non-Java read access

See doc/native-interface.md.

dossfs /doss-devel /mnt
grep -i endeavour /mnt/962b6910
convert /mnt/014908b0 /tmp/kangaroo.jpg

Building

Requirements:

  • Java 8
  • Maven 3

Compile by typing:

mvn package

To build both the server and remote client:

cd project
mvn package

License

Copyright 2013 National Library of Australia

DOSS 2 is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0:

http://www.apache.org/licenses/LICENSE-2.0

See the file LICENSE for details.