Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Neo4J Support #320

Merged
merged 27 commits into from
Jan 15, 2020
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ cli/migrate
.godoc.pid
vendor/
.vscode/
.idea/
5 changes: 5 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ before_install:
# Install golangci-lint
- curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.22.2
- echo "TRAVIS_GO_VERSION=${TRAVIS_GO_VERSION}"
# Install seabolt for Neo4j
- sudo apt-get install -y libssl1.0.0
- wget https://github.com/neo4j-drivers/seabolt/releases/download/v1.7.4/seabolt-1.7.4-Linux-ubuntu-$(lsb_release -rs).deb
- sudo dpkg -i seabolt-1.7.4-Linux-ubuntu-$(lsb_release -rs).deb
- rm seabolt-1.7.4-Linux-ubuntu-$(lsb_release -rs).deb

install:
- go get github.com/mattn/goveralls
Expand Down
10 changes: 9 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,20 @@ ARG VERSION

RUN apk add --no-cache git gcc musl-dev

# dependencies for neo4j
RUN apk add --update --no-cache ca-certificates cmake make g++ libressl-dev openssl-dev git curl pkgconfig
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird that adding libressl-dev works and using only openssl-dev or libressl-dev doesn't...

It looks like Alpine Linux uses OpenSSL by default since 3.9 so we probably shouldn't be using LibreSSL.

Theoretically, seabolt builds with OpenSSL on Alpine Linux 3.9. Not sure what's different with Alpine Linux 3.11

Let's find the correct variables and values to set so we can use OpenSSL. See:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it looks like the built seabolt library needs to be copied over to the final Docker image:

$ docker run --rm migrate/migrate:mvid
Error loading shared library libseabolt17.so.1: No such file or directory (needed by /migrate)
<OUTPUT TRUCATED>
$

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding COPY --from=downloader /usr/local/lib/libseabolt* /lib/ fixes the issue. I'd this before COPY --from=downloader /go/src/github.com/golang-migrate/migrate/build/migrate.linux-386 /migrate


# build seabolt for neo4j driver
RUN git clone -b 1.7 https://github.com/neo4j-drivers/seabolt.git /seabolt
WORKDIR /seabolt/build
RUN cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_LIBDIR=lib .. && cmake --build . --target install

WORKDIR /go/src/github.com/golang-migrate/migrate

COPY . ./

ENV GO111MODULE=on
ENV DATABASES="postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver firebird"
ENV DATABASES="postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver firebird neo4j"
ENV SOURCES="file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab"

RUN go build -a -o build/migrate.linux-386 -ldflags="-s -w -X main.Version=${VERSION}" -tags "$DATABASES $SOURCES" ./cmd/migrate
Expand Down
13 changes: 7 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
SOURCE ?= file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver firebird
DATABASE_TEST ?= $(DATABASE) sqlite neo4j
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
REPO_OWNER ?= $(shell cd .. && basename "$$(pwd)")
Expand All @@ -8,11 +9,11 @@ COVERAGE_DIR ?= .coverage

build-cli: clean
-mkdir ./cli/build
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o ../../cli/build/migrate.linux-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -a -o ../../cli/build/migrate.linux-armv7 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -o ../../cli/build/migrate.linux-arm64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -a -o ../../cli/build/migrate.darwin-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -a -o ../../cli/build/migrate.windows-amd64.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
# cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o ../../cli/build/migrate.linux-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
dhui marked this conversation as resolved.
Show resolved Hide resolved
# cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -a -o ../../cli/build/migrate.linux-armv7 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
# cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -o ../../cli/build/migrate.linux-arm64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=1 GOOS=darwin GOARCH=amd64 go build -a -o ../../cli/build/migrate.darwin-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
# cd ./cmd/migrate && CGO_ENABLED=1 GOOS=windows GOARCH=amd64 go build -a -o ../../cli/build/migrate.windows-amd64.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cli/build && find . -name 'migrate*' | xargs -I{} tar czf {}.tar.gz {}
cd ./cli/build && shasum -a 256 * > sha256sum.txt
cat ./cli/build/sha256sum.txt
Expand All @@ -34,7 +35,7 @@ test:

test-with-flags:
@echo SOURCE: $(SOURCE)
@echo DATABASE: $(DATABASE)
@echo DATABASE_TEST: $(DATABASE_TEST)

@go test $(TEST_FLAGS) ./...

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Database drivers run migrations. [Add a new database?](database/driver.go)
* [Cassandra](database/cassandra)
* [SQLite](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [MySQL/ MariaDB](database/mysql)
* [Neo4j](database/neo4j) ([todo #167](https://github.com/mattes/migrate/issues/167))
* [Neo4j](database/neo4j)
* [MongoDB](database/mongodb)
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
Expand Down
7 changes: 4 additions & 3 deletions database/driver.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ import (
)

var (
ErrLocked = fmt.Errorf("can't acquire lock")
ErrLocked = fmt.Errorf("can't acquire lock")
ErrNotLocked = fmt.Errorf("can't unlock, as not currently locked")
)

const NilVersion int = -1
Expand All @@ -33,7 +34,7 @@ var drivers = make(map[string]Driver)
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a migrate/cli/build_<driver-name>.go file
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
//
// Guidelines:
Expand Down Expand Up @@ -61,7 +62,7 @@ type Driver interface {
// all migrations have been run.
Unlock() error

// Run applies a migration to the database. migration is garantueed to be not nil.
// Run applies a migration to the database. migration is guaranteed to be not nil.
Run(migration io.Reader) error

// SetVersion saves version and dirty state.
Expand Down
11 changes: 11 additions & 0 deletions database/neo4j/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# neo4j
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add docs about installing seabolt and statically linking OpenSSL to the README


`neo4j://user:password@host:port/`

| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `user` | Contained within `AuthConfig` | The user to sign in as |
| `password` | Contained within `AuthConfig` | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 7687) |
| | `MigrationsLabel` | Name of the migrations node label |
97 changes: 97 additions & 0 deletions database/neo4j/TUTORIAL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
## Create migrations
Let's create nodes called `Users`:
```
migrate create -ext cypher -dir db/migrations -seq create_user_nodes
```
If there were no errors, we should have two files available under `db/migrations` folder:
- 000001_create_user_nodes.down.cypher
- 000001_create_user_nodes.up.cypher

Note the `cypher` extension that we provided.

In the `.up.cypher` file let's create the table:
```
CREATE (u1:User {name: "Peter"})
CREATE (u2:User {name: "Paul"})
CREATE (u3:User {name: "Mary"})
```
And in the `.down.sql` let's delete it:
```
MATCH (u:User) WHERE u.name IN ["Peter", "Paul", "Mary"] DELETE u
```
Ideally your migrations should be idempotent. You can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)

## Run migrations
```
migrate -database ${NEO4J_URL} -path db/migrations up
```
Let's check if the table was created properly by running `bin/cypher-shell -u neo4j -p password`, then `neo4j> MATCH (u:User)`
The output you are supposed to see:
```
+-----------------------------------------------------------------+
| u |
+-----------------------------------------------------------------+
| (:User {name: "Peter") |
| (:User {name: "Paul") |
| (:User {name: "Mary") |
+-----------------------------------------------------------------+
```
Great! Now let's check if running reverse migration also works:
```
migrate -database ${NEO4J_URL} -path db/migrations down
```
Make sure to check if your database changed as expected in this case as well.

## Database transactions

To show database transactions usage, let's create another set of migrations by running:
```
migrate create -ext cypher -dir db/migrations -seq add_mood_to_users
```
Again, it should create for us two migrations files:
- 000002_add_mood_to_users.down.cypher
- 000002_add_mood_to_users.up.cypher

In Neo4j, when we want our queries to be done in a transaction, we need to wrap it with `:BEGIN` and `:COMMIT` commands.
Migration up:
```
:BEGIN

MATCH (u:User)
SET u.mood = "Cheery"

:COMMIT
```
Migration down:
```
:BEGIN

MATCH (u:User)
SET u.mood = null

:COMMIT
```

## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
import (
"log"

"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/neo4j"
_ "github.com/golang-migrate/migrate/v4/source/file"
)

func main() {
m, err := migrate.New(
"file://db/migrations",
"neo4j://neo4j:password@localhost:7687/")
if err != nil {
log.Fatal(err)
}
if err := m.Up(); err != nil {
log.Fatal(err)
}
}
```
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DROP CONSTRAINT ON (m:Movie) ASSERT m.Name IS UNIQUE
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CREATE CONSTRAINT ON (m:Movie) ASSERT m.Name IS UNIQUE
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
MATCH (m:Movie)
DELETE m
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
CREATE (:Movie {name: "Footloose"})
CREATE (:Movie {name: "Ghost"})
Loading