dataset is a command line tool, Go package, shared library and Python package for working with JSON objects as collections on disc, in an S3 bucket or in Cloud Storage
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
analyzers/allwords
assets
cmd/dataset
css
demos
docs
etc
gsheets
how-to
libdataset
py
reference
scripts
tbl
.gitignore
DEVELOPMENT-NOTES.md
INSTALL.md
LICENSE
Makefile
NOTES.md
README.md
RELEASES.md
TODO.md
attachments.go
attachments_test.go
blobstore_test.go
bucketnames.go
buckets.go
buckets_test.go
codemeta.json
dataset.go
dataset_test.go
doc.go
formats.go
frame.go
frame_test.go
grid.go
grid_test.go
index.html
install.html
license.html
main_test.go
mk-website.bash
name-as-text.html
name-as-text.md
nav.md
package-versions.bash
page.tmpl
pairtree.go
pairtree_test.go
publish.bash
rebuild-assets.bash
repair.go
repair_test.go
search.go
search_test.go
sort.go
sync.go
sync_test.go
test_cmd.bash
update_version.py

README.md

dataset DOI

dataset is a command line tool, Go package, and an experimental C shared library for working with JSON objects as collections. Collections can be stored on disc or in Cloud Storage. JSON objects are stored in collections as plain UTF-8 text. This means the objects can be accessed with common Unix text processing tools as well as most programming languages.

The dataset command line tool supports common data manage operations such as initialization of collections, creation, reading, updating and deleting JSON objects in the collection. Some of its enhanced features include the ability to generate data frames as well as the ability to import and export JSON objects to and from CSV files and Google Sheets.

dataset is written in the Go programming language. It can be used as a Go package by other Go based software. Go supports generating C shared libraries. By compiling the Go source you can create a libdataset C shared library. The C shared library is currently being used by the DLD Group in Caltech Library experimentally from Python 3. This approach looks promising to support other languages (e.g. Julia can easily use dataset via its ccall function, while R, Octave and NodeJS would probably need some C++ wrapping code).

See getting-started-with-datataset.md for a tour and tutorial.

Design choices

dataset isn't a database or a replacement for repository systems. It is guided by the idea that you should be able to work with text files, the JSON objects documents, with standard Unix text utilities. It is intended to be simple to use with minimal setup (e.g. dataset init mycollection.ds would create a new collection called 'mycollection.ds'). It is built around a few abstractions -- dataset stores JSON objects in collections, collections are a folder(s) containing the JSON object documents and any attachments, a collections.json file describes the mapping of keys to folder locations). dataset takes minimal system resources and keeps all content, except JSON object attachments, in plain UTF-8 text. Attachments are stored using the venerable "tar" archive format.

The choice of plain UTF-8 and tar balls is intended to help future proof reading dataset collections. Care has been taken to keep dataset simple enough and light weight enough that it will run on a machine as small as a Raspberry Pi while being equally comfortable on a more resource rich server or desktop environment. It should be easy to do alternative implementations in any language that has good string, JSON support and memory management.

Workflows

A typical library processing pattern is to write a "harvester" which then stores it results in a dataset collection. Write something that transforms or aggregates harvested options and then write a final rendering program to prepare the data for the web. The the hearvesters are typically written in Python or as a simple Bash script storing the results in dataset. Depending on the performance needs our transform and aggregates stage are written either in Python or Go and our final rendering stages are typically written in Python or as simple Bash scripts.

Features

dataset supports

You can work with dataset collections via the command line tool, via Go using the dataset package or in Python 3.7 using a python package. dataset is useful for general data science applications which need intermediate JSON object management but not a full blown database.

Limitations of dataset

dataset has many limitations, some are listed below

  • it is not a multi-process, multi-user data store (it's files on "disc" without locking)
  • it is not a replacement for a repository management system
  • it is not a general purpose database system
  • it does not supply version control on collections or objects

Read next ...

Explore dataset through A Shell Example, Getting Started with Dataset, How To guides, topics and Documentation.

Releases

Compiled versions are provided for Linux (amd64), Mac OS X (amd64), Windows 10 (amd64) and Raspbian (ARM7). See https://github.com/caltechlibrary/dataset/releases.