Skip to content
Go framework to simplify CRUD of structured data using Graph operations
Go
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
drivers Implement setting from offset to support pagination Sep 30, 2015
example Finished work on both AND and OR query interfaces. Memsearch modified… Sep 9, 2015
examples/social Finished work on both AND and OR query interfaces. Memsearch modified… Sep 9, 2015
indexer
presentation Fix function comments based on best practices from Effective Go Feb 18, 2019
req bug fix: NewContext spelling mistake. And add usage_test.go. Aug 19, 2015
res
search Implement setting from offset to support pagination Sep 30, 2015
store Fix function comments based on best practices from Effective Go Feb 18, 2019
testx update store_test to reflect source->modifier change Oct 26, 2015
x Make json output by social.go pretty and readable. Avoid debug loggin… Aug 12, 2015
.gitignore add omnigraffle doc and images. Modify README Jun 27, 2015
LICENSE Initial commit Jun 19, 2015
README.md tell about IRC channel Nov 3, 2015
allthings.jpg Gocrud all things, thanks Monish Aug 27, 2015
datastore.md Allow username password auth in Cassandra driver Aug 13, 2015
example.md Move NewUpdate and NewQuery interfaces to store package, which is whe… Aug 12, 2015
usage.go fix for godoc Aug 19, 2015
usage_test.go
wercker.yml Set package-dir to allow local build Oct 8, 2015

README.md

gocrud

Go framework to simplify creating, reading, updating, and deleting arbitrary depth structured data — to make building REST services fast and easy.

wercker status GoDoc

Releases

Note that the master branch always refers to latest version of Gocrud, which would contain breaking changes. To use stable version of Gocrud APIs, please use the packages released via gopkg.in.

Gocrud version Install instructions Godoc Source
v1 (stable) go get -v gopkg.in/manishrjain/gocrud.v1/... godoc source
master (dev) go get -v github.com/manishrjain/gocrud/... godoc source

Questions / Support

I primarily use IRC on freenode network. Channel is #gocrud. I'm mrjn on the network. #gocrud on freenode

I also hang out at the gophers.slack.com, at #gocrud channel. Although, Slack doesn't have a linux client which is what my workstation runs, and hence, I prefer IRC over Slack. You can get an invitation to join Slack via this link: Gopher Slack Signup. You can also direct message me, my user id is @manishrjain.

Why?

Courtesy: Monish, co-founder karma.wiki

Having built over 3 different startup backends, I think a lot of time is wasted figuring out and coding CRUD for data structures. In addition, the choice of database has to be made up front, which causes a lot of headache for startup founders. Gocrud was written with the aim to make CRUD easy, and provide the flexibility to switch out both the underlying storage and search engines at any stage of development.

Data stores

Datastore Driver Available Status
LevelDB Yes Ready
MySQL Yes Needs to implement Iterate func
PostgreSQL Yes Needs to implement Iterate func
Cassandra Yes Ready
MongoDB Yes Needs to implement Iterate func
Google Datastore Yes Needs to implement Iterate func
RethinkDB Yes Needs to implement Iterate func
Amazon DynamoDB No Needs work
Datastore usage shows how to use and initialize various datastores. One can add support for more by implementing this interface:
type Store interface {
  Init(args ...string)
  Commit(its []*x.Instruction) error
  IsNew(subject string) bool
  GetEntity(subject string) ([]x.Instruction, error)
  Iterate(fromId string, num int, ch chan x.Entity) (int, error)
}

Search engines

Search Engine Drive Available
Elastic Search Yes
Solr No

Can be added by implementing these interfaces:

type Engine interface {
	Init(args ...string)
	Update(x.Doc) error
	NewQuery(kind string) Query
}

type Query interface {
	Limit(num int) Query
	Order(field string) Query
	Run() ([]x.Doc, error)
  // and few others
}

Framework

This framework is built to follow these principles:

  1. Versioning: Keep track of all edits to the data, including deletion operations.
  2. Authorship: Be able to track who edited (/deleted) what.
  3. Retention: On deletion, only mark it as deleted. Never actually delete any data.

The framework makes it easy to have Parent-Child relationships, quite common in today’s CRUD operations. For e.g.

- Posts created by User (User -> Post)
- Comments on Posts (Post -> Comment)
- Likes on Posts (Post -> Like)
- Likes on Comments (Comment -> Like)

And be able to traverse these relationships and retrieve all of the children, grandchildren etc. For e.g. (User -> Post -> [(Comment -> Like), Like])

The framework does this by utilizing Graph operations, but without using a Graph database. This means the framework can be used to quickly build a Go backend to serve arbitrarily complex data, while still using your database of choice. See example usage

Dependency management

Users who import Gocrud into their packages are responsible to organize and maintain all of their dependencies to ensure code compatibility and build reproducibility. Gocrud makes no direct use of dependency management tools like Godep.

Performance considerations

For the example, this is what gets stored in the database:

mysql> select * from instructions;
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
| subject_id | subject_type | predicate | object                               | object_id | nano_ts             | source  | id |
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
| uid_oNM    | User         | Post      | NULL                                 | wClGp     | 1435408916326573229 | uid_oNM |  1 |
| wClGp      | Post         | body      | "You can search for cat videos here" |           | 1435408916326573229 | uid_oNM |  2 |
| wClGp      | Post         | tags      | ["search","cat","videos"]            |           | 1435408916326573229 | uid_oNM |  3 |
| wClGp      | Post         | url       | "www.google.com"                     |           | 1435408916326573229 | uid_oNM |  4 |
| wClGp      | Post         | Like      | NULL                                 | kStx9     | 1435408916341828408 | uid_qB3 |  5 |
| kStx9      | Like         | thumb     | 1                                    |           | 1435408916341828408 | uid_qB3 |  6 |
| wClGp      | Post         | Comment   | NULL                                 | 8f78r     | 1435408916341828408 | uid_qB3 |  7 |
| 8f78r      | Comment      | body      | "Comment by on the post"             |           | 1435408916341828408 | uid_qB3 |  8 |
| wClGp      | Post         | Like      | NULL                                 | Gyd7G     | 1435408916352622582 | uid_a30 |  9 |
| Gyd7G      | Like         | thumb     | 1                                    |           | 1435408916352622582 | uid_a30 | 10 |
| 8f78r      | Comment      | Like      | NULL                                 | q2IKK     | 1435408916357443075 | uid_I5u | 11 |
| q2IKK      | Like         | thumb     | 1                                    |           | 1435408916357443075 | uid_I5u | 12 |
| 8f78r      | Comment      | Comment   | NULL                                 | g8llL     | 1435408916357443075 | uid_I5u | 13 |
| g8llL      | Comment      | body      | "Comment xv on comment"              |           | 1435408916357443075 | uid_I5u | 14 |
| q2IKK      | Like         | Comment   | NULL                                 | oaztb     | 1435408916368908590 | uid_SPX | 15 |
| oaztb      | Comment      | body      | "Comment kL on Like"                 |           | 1435408916368908590 | uid_SPX | 16 |
| 8f78r      | Comment      | censored  | true                                 |           | 1435408916377065650 | uid_D2g | 17 |
| kStx9      | Like         | _delete_  | true                                 |           | 1435408916384422689 | uid_2a5 | 18 |
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
18 rows in set (0.00 sec)

The writes are in constant time, where each (entity,predicate) constitutes one row. As the properties per entity grow, more rows need to be read (1 row = 1 edge/predicate) to get the entity, it's predicates and it's children. This however, shouldn't be much of a concern for any standard data, which has limited number of predicates/properties per entity. Gocrud in addition, retrieves all children in parallel via goroutines, instead of retrieving them one by one.

Property value filtering, sorting, full and partial text matching are now being made available via various search engines. Gocrud provides a search interface, which provides the most common search functionality right out of the box. Thus, there's a clear distinction between data store and search right from the beginning.

Reserved keywords

The following predicates are reserved by the framework, and shouldn't be used by the caller. Currently, this guideline isn't being hardly enforced by the framework.

Predicate Meaning
_parent_ Stores an edge from child -> parent entity.
_delete_ Marks a particular entity as deleted.

Contact

Feel free to contact me at my Twitter handle @manishrjain for any discussions related to this framework. Also, feel free to send pull requests, they're welcome!

You can’t perform that action at this time.