Metadata DB for new archive project
Clone or download
Latest commit ba32386 Dec 3, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Godeps missing dependencies Jul 26, 2018
api Health - fix for postgres gone not detected properly Dec 4, 2018
batch Import/Export related stuff May 16, 2018
bindata Autoname - initial dirty commit. WIP Jun 2, 2017
cmd fix for typo in emitter Nov 21, 2018
data german auto names (#40) Dec 4, 2018
docs Merge from master, take models from master. May 4, 2017
events fix for typo in emitter Nov 21, 2018
exporter Importer - kab.co.il tv shows name and descriptions in Hebrew May 22, 2018
hebcal Hebrew Calendar - first commit Nov 9, 2017
importer fix for typo in emitter Nov 21, 2018
migrations Importer - blog, full revisit of making relative links and post filte… Jul 26, 2018
misc misc - yet another fix to importer scripts (logs cleanup) Aug 8, 2018
models Importer - blog, full revisit of making relative links and post filte… Jul 26, 2018
permissions Auth - skip token verification when authentiction is not enabled Aug 13, 2018
storage Storage import - fix for sql error on deleting storages Feb 8, 2018
utils Fix breaking test, disable go 1.10 cache in `make test` and remove so… Jun 5, 2018
vendor missing dependencies Jul 26, 2018
version bump version Dec 4, 2018
.codeclimate.json Setup for CodeClimate was added Dec 27, 2017
.gitignore Importer - blog (mainly laitman.ru but with preparation for other lan… Jul 22, 2018
LICENSE add license Dec 18, 2017
Makefile Fix breaking test, disable go 1.10 cache in `make test` and remove so… Jun 5, 2018
README.md Setup for CodeClimate was added Dec 27, 2017
config.sample.toml Importer - blog (mainly laitman.ru but with preparation for other lan… Jul 22, 2018
main.go Setup - use viper for config. WIP Nov 2, 2016
queries.sql Importer - blog, full revisit of making relative links and post filte… Jul 26, 2018
sqlboiler.sample.toml Move sqlboiler to sample. Mar 6, 2017

README.md

MDB - Metadata DB

Maintainability

Test Coverage

Overview

BB archive Metadata Database.

This system aims to be a single source of truth for all content produced by Bnei Baruch.

Commands

The mdb is meant to be executed as command line. Type mdb <command> -h to see how to use each command.

mdb server

Run the server

mdb config <path>

Generate default configuration in the given path. If path is omitted STDOUT is used instead. Note that default value to config file is config.toml in project root directory.

 mdb migration my-migration-name

Create new migration. (See Schema migrations section for more information).

mdb version

Print the version of MDB

Implementation Notes

Dates and Times

All timestamps fields are expecting values in UTC only.

Languages

Languages are represented in the system as a two letters code adhering to the ISO_639-1 standard.

Special values:

  • Unknown - xx
  • Multiple languages - zz

Release and Deployment

Once development is done, all tests are green, we want to go live. All we have to do is simply execute misc/release.sh.

To add a pre-release tag, add the relevant environment variable. For example,

PRE_RELEASE=rc.1 misc/release.sh

Schema Migrations

We keep track of all changes to the MDB schema under migrations. These are pure postgres sql scripts. To create a new migration file with name run in project root directory:

mdb migration my-migration-name

This will create a migration file in migrations directory with name like: 2017-01-07_14:21:02_my-migration-name.sql

They play along well with rambler A simple and language-independent SQL schema migration tool. Download the rambler executable for your system from the release page. (on linux chmod +x)

Under migrations folder add a rambler.json config file. An example:

{
  "driver": "postgresql",
  "protocol": "tcp",
  "host": "localhost",
  "port": 5432,
  "user": "",
  "password": "",
  "database": "mdb",
  "directory": "migrations",
  "table": "migrations"
}

Important make sure never to commit such files to SCM.

On the command line:

rambler -c migrations/rambler.json apply -a

Logging

We use logrus for logging.

All logs are written to STDOUT or STDERR. It's up to the running environment to pipe these into physical files and rotate those using logrotate(8).

Rollbar

If the rollbar-token config is found we'll use our own recovery middleware to send errors to Rollbar. If not, we'll use gin.Recovery() to print stacktrace to console. Using rollbar is meant for production environment.

In addition, you could log whatever error you want to rollbar directly, for example:

   if _, err := SomeErrorProneFunc(); err != nil {
       rollbar.Error("level", err,...)
   }

Check out the docs for more info on how to use the Rollbar client.

Documentation

Documentation is based on tests and will be generated automatically with each make build. To generate static html documentation (docs.html) install:

npm install -g aglio

Then run:

make api

Installation details

Postgresql installation

https://wiki.postgresql.org/wiki/Apt

Go related installations

sudo apt-get update
sudo curl -O https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz

Detailes can be found here: https://www.digitalocean.com/community/tutorials/how-to-install-go-1-6-on-ubuntu-14-04)

sudo tar -xvf go1.8.linux-amd64.tar.gz
sudo mv go /usr/local
export GOROOT=/user/local/go

While at /home/kolmanv/go

export GOPATH=/home/kolmanv/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
git clone https://github.com/Bnei-Baruch/mdb.git

Install Packages - Using godep

go get -u github.com/jteeuwen/go-bindata/...
go get gopkg.in/gin-gonic/gin.v1
go get github.com/lib/pq
go get github.com/tools/godep
# https://github.com/tools/godep
godep save

License

MIT