Skip to content

Commit

Permalink
Upgrade the SQLCipher to the latest version.
Browse files Browse the repository at this point in the history
  • Loading branch information
xeodou committed May 4, 2020
2 parents e2b0cbb + 61ad8da commit 40fb371
Show file tree
Hide file tree
Showing 57 changed files with 15,179 additions and 8,931 deletions.
8 changes: 8 additions & 0 deletions .github/FUNDING.yml
@@ -0,0 +1,8 @@
# These are supported funding model platforms

github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: mattn # Replace with a single Patreon username
open_collective: mattn # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
custom: # Replace with a single custom sponsorship URL
5 changes: 3 additions & 2 deletions .travis.yml
Expand Up @@ -11,6 +11,8 @@ go:
- 1.9.x
- 1.10.x
- 1.11.x
- 1.12.x
- 1.13.x
- master

before_install:
Expand All @@ -25,6 +27,5 @@ before_install:
script:
- $HOME/gopath/bin/goveralls -repotoken PfqH9iFyzW3daUxflzllSougjTxvFwQZE
- go test -race -v . -tags ""
- go test -race -v . -tags "libsqlite3"
- go test -race -v . -tags "sqlite_allow_uri_authority sqlite_app_armor sqlite_foreign_keys sqlite_fts5 sqlite_icu sqlite_introspect sqlite_json sqlite_secure_delete sqlite_see sqlite_stat4 sqlite_trace sqlite_userauth sqlite_vacuum_incr sqlite_vtable sqlite_unlock_notify"
- go test -race -v . -tags "sqlite_allow_uri_authority sqlite_app_armor sqlite_foreign_keys sqlite_fts5 sqlite_icu sqlite_introspect sqlite_json sqlite_preupdate_hook sqlite_secure_delete sqlite_see sqlite_stat4 sqlite_trace sqlite_vacuum_incr sqlite_vtable sqlite_unlock_notify"
- go test -race -v . -tags "sqlite_vacuum_full"
115 changes: 101 additions & 14 deletions README.md
Expand Up @@ -4,12 +4,20 @@ go-sqlcipher

SQLCipher driver conforming to the built-in database/sql interface and using the latest sqlite3 code.

[![GoDoc Reference](https://godoc.org/github.com/xeodou/go-sqlcipher?status.svg)](http://godoc.org/github.com/xeodou/go-sqlcipher)
[![Build Status](https://travis-ci.org/xeodou/go-sqlcipher.svg?branch=master)](https://travis-ci.org/xeodou/go-sqlcipher)
[![Coverage Status](https://coveralls.io/repos/xeodou/go-sqlcipher/badge.svg?branch=master)](https://coveralls.io/r/xeodou/go-sqlcipher?branch=master)
[![Go Report Card](https://goreportcard.com/badge/github.com/xeodou/go-sqlcipher)](https://goreportcard.com/report/github.com/xeodou/go-sqlcipher)

NOTE: v2.0.1 or higher is unfortunatal release. So there are no big changes. And does not provide v2 feature.

# Description

which is
`3.26.0`
`3.31.0`

Working with sqlcipher version which is
`4.0.1`
`4.3.0`

It's wrapper with
* [go-sqlite3](https://github.com/mattn/go-sqlite3) sqlite3 driver for go that using database/sql.
Expand All @@ -34,15 +42,21 @@ To upgrade SQLCipher from 3.x to 4.x, please take a look of:

### Overview

- [go-sqlite3](#go-sqlite3)
- [Description](#description)
- [Overview](#overview)
- [Installation](#installation)
- [API Reference](#api-reference)
- [Connection String](#connection-string)
- [DSN Examples](#dsn-examples)
- [Features](#features)
- [Usage](#usage)
- [Feature / Extension List](#feature--extension-list)
- [Compilation](#compilation)
- [Android](#android)
- [ARM](#arm)
- [Cross Compile](#cross-compile)
- [Google Cloud Platform](#google-cloud-platform)
- [ARM](#arm)
- [Cross Compile](#cross-compile)
- [Google Cloud Platform](#google-cloud-platform)
- [Linux](#linux)
- [Alpine](#alpine)
- [Fedora](#fedora)
Expand All @@ -52,11 +66,22 @@ To upgrade SQLCipher from 3.x to 4.x, please take a look of:
- [Errors](#errors)
- [User Authentication](#user-authentication)
- [Compile](#compile)
- [Usage](#usage)
- [Usage](#usage-1)
- [Create protected database](#create-protected-database)
- [Password Encoding](#password-encoding)
- [Available Encoders](#available-encoders)
- [Restrictions](#restrictions)
- [Support](#support)
- [User Management](#user-management)
- [SQL](#sql)
- [Examples](#examples)
- [*SQLiteConn](#sqliteconn)
- [Attached database](#attached-database)
- [Extensions](#extensions)
- [Spatialite](#spatialite)
- [FAQ](#faq)
- [License](#license)
- [Author](#author)

# Installation

Expand Down Expand Up @@ -134,6 +159,9 @@ This package allows additional configuration of features available within SQLite

[Click here for more information about build tags / constraints.](https://golang.org/pkg/go/build/#hdr-Build_Constraints)

**Please notice**
The `userAuthentication` extention is not support the library, since the SQLCipher is already let you create the encrypted database.

### Usage

If you wish to build this library with additional extensions / features.
Expand All @@ -145,7 +173,7 @@ go build --tags "<FEATURE>"

If you want to build the project without the `libcrypto`, you could specific the openssl library by using the command.
```bash
CGO_ENABLE=0 CGO_LDFLAGS="-L/usr/local/opt/openssl/lib" CGO_CPPFLAGS="-I/usr/local/opt/openssl/include" go build build _example/encrypto/encrypto.go
CGO_ENABLE=1 CGO_LDFLAGS="-L/usr/local/opt/openssl/lib" CGO_CPPFLAGS="-I/usr/local/opt/openssl/include" go build _example/encrypto/encrypto.go
```

For available features see the extension list.
Expand All @@ -172,6 +200,7 @@ go build --tags "icu json1 fts5 secure_delete"
| International Components for Unicode | sqlite_icu | This option causes the International Components for Unicode or "ICU" extension to SQLite to be added to the build |
| Introspect PRAGMAS | sqlite_introspect | This option adds some extra PRAGMA statements. <ul><li>PRAGMA function_list</li><li>PRAGMA module_list</li><li>PRAGMA pragma_list</li></ul> |
| JSON SQL Functions | sqlite_json | When this option is defined in the amalgamation, the JSON SQL functions are added to the build automatically |
| Pre Update Hook | sqlite_preupdate_hook | Registers a callback function that is invoked prior to each INSERT, UPDATE, and DELETE operation on a database table. |
| Secure Delete | sqlite_secure_delete | This compile-time option changes the default setting of the secure_delete pragma.<br><br>When this option is not used, secure_delete defaults to off. When this option is present, secure_delete defaults to on.<br><br>The secure_delete setting causes deleted content to be overwritten with zeros. There is a small performance penalty since additional I/O must occur.<br><br>On the other hand, secure_delete can prevent fragments of sensitive information from lingering in unused parts of the database file after it has been deleted. See the documentation on the secure_delete pragma for additional information |
| Secure Delete (FAST) | sqlite_secure_delete_fast | For more information see [PRAGMA secure_delete](https://www.sqlite.org/pragma.html#pragma_secure_delete) |
| Tracing / Debug | sqlite_trace | Activate trace functions |
Expand Down Expand Up @@ -270,7 +299,7 @@ Required dependency
brew install sqlite3
```

For OSX there is an additional package install which is required if you whish to build the `icu` extension.
For OSX there is an additional package install which is required if you wish to build the `icu` extension.

This additional package can be installed with `homebrew`.

Expand Down Expand Up @@ -460,6 +489,16 @@ If you want your own extension to be listed here or you want to add a reference
Spatialite is available as an extension to SQLite, and can be used in combination with this repository.
For an example see [shaxbee/go-spatialite](https://github.com/shaxbee/go-spatialite).

## extension-functions.c from SQLite3 Contrib

extension-functions.c is available as an extension to SQLite, and provides the following functions:

- Math: acos, asin, atan, atn2, atan2, acosh, asinh, atanh, difference, degrees, radians, cos, sin, tan, cot, cosh, sinh, tanh, coth, exp, log, log10, power, sign, sqrt, square, ceil, floor, pi.
- String: replicate, charindex, leftstr, rightstr, ltrim, rtrim, trim, replace, reverse, proper, padl, padr, padc, strfilter.
- Aggregate: stdev, variance, mode, median, lower_quartile, upper_quartile

For an example see [dinedal/go-sqlite3-extension-functions](https://github.com/dinedal/go-sqlite3-extension-functions).

# FAQ

- Getting insert error while query is opened.
Expand All @@ -484,15 +523,19 @@ For an example see [shaxbee/go-spatialite](https://github.com/shaxbee/go-spatial

Why is it racy if I use a `sql.Open("sqlite3", ":memory:")` database?

Each connection to :memory: opens a brand new in-memory sql database, so if
Each connection to `":memory:"` opens a brand new in-memory sql database, so if
the stdlib's sql engine happens to open another connection and you've only
specified ":memory:", that connection will see a brand new database. A
workaround is to use "file::memory:?mode=memory&cache=shared". Every
specified `":memory:"`, that connection will see a brand new database. A
workaround is to use `"file::memory:?cache=shared"` (or `"file:foobar?mode=memory&cache=shared"`). Every
connection to this string will point to the same in-memory database.

Note that if the last database connection in the pool closes, the in-memory database is deleted. Make sure the [max idle connection limit](https://golang.org/pkg/database/sql/#DB.SetMaxIdleConns) is > 0, and the [connection lifetime](https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime) is infinite.

For more information see
* [#204](https://github.com/mattn/go-sqlite3/issues/204)
* [#511](https://github.com/mattn/go-sqlite3/issues/511)
* https://www.sqlite.org/sharedcache.html#shared_cache_and_in_memory_databases
* https://www.sqlite.org/inmemorydb.html#sharedmemdb

- Reading from database with large amount of goroutines fails on OSX.

Expand All @@ -507,12 +550,56 @@ For an example see [shaxbee/go-spatialite](https://github.com/shaxbee/go-spatial

You need to implement the feature or call the sqlite3 cli.

* Print some waring messages like `warning: 'RAND_add' is deprecated: first deprecated in OS X 10.7`
More information see [#305](https://github.com/mattn/go-sqlite3/issues/305)

- Error: `database is locked`

You can ignore these messages.

License
-------
Example:
```go
db, err := sql.Open("sqlite3", "file:locked.sqlite?cache=shared")
```

Second please set the database connections of the SQL package to 1.

```go
db.SetMaxOpenConns(1)
```

More information see [#209](https://github.com/mattn/go-sqlite3/issues/209)

## Contributors

### Code Contributors

This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
<a href="https://github.com/mattn/go-sqlite3/graphs/contributors"><img src="https://opencollective.com/mattn-go-sqlite3/contributors.svg?width=890&button=false" /></a>

### Financial Contributors

Become a financial contributor and help us sustain our community. [[Contribute](https://opencollective.com/mattn-go-sqlite3/contribute)]

#### Individuals

<a href="https://opencollective.com/mattn-go-sqlite3"><img src="https://opencollective.com/mattn-go-sqlite3/individuals.svg?width=890"></a>

#### Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [[Contribute](https://opencollective.com/mattn-go-sqlite3/contribute)]

<a href="https://opencollective.com/mattn-go-sqlite3/organization/0/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/0/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/1/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/1/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/2/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/2/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/3/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/3/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/4/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/4/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/5/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/5/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/6/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/6/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/7/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/7/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/8/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/8/avatar.svg"></a>
<a href="https://opencollective.com/mattn-go-sqlite3/organization/9/website"><img src="https://opencollective.com/mattn-go-sqlite3/organization/9/avatar.svg"></a>

# License

MIT:

Expand Down
8 changes: 4 additions & 4 deletions _example/limit/limit.go
Expand Up @@ -27,7 +27,7 @@ func createBulkInsertQuery(n int, start int) (query string, args []interface{})
return
}

func bukInsert(db *sql.DB, query string, args []interface{}) (err error) {
func bulkInsert(db *sql.DB, query string, args []interface{}) (err error) {
stmt, err := db.Prepare(query)
if err != nil {
return
Expand Down Expand Up @@ -76,7 +76,7 @@ func main() {

num := 400
query, args := createBulkInsertQuery(num, 0)
err = bukInsert(db, query, args)
err = bulkInsert(db, query, args)
if err != nil {
log.Fatal(err)
}
Expand All @@ -88,7 +88,7 @@ func main() {
log.Printf("updated SQLITE_LIMIT_VARIABLE_NUMBER: %d", limitVariableNumber)

query, args = createBulkInsertQuery(num, num)
err = bukInsert(db, query, args)
err = bulkInsert(db, query, args)
if err != nil {
if err != nil {
log.Printf("expect failed since SQLITE_LIMIT_VARIABLE_NUMBER is too small: %v", err)
Expand All @@ -102,7 +102,7 @@ func main() {
log.Printf("updated SQLITE_LIMIT_VARIABLE_NUMBER: %d", limitVariableNumber)

query, args = createBulkInsertQuery(500, num+num)
err = bukInsert(db, query, args)
err = bulkInsert(db, query, args)
if err != nil {
if err != nil {
log.Fatal(err)
Expand Down
17 changes: 11 additions & 6 deletions _example/mod_regexp/Makefile
@@ -1,22 +1,27 @@
ifeq ($(OS),Windows_NT)
EXE=extension.exe
EXT=sqlite3_mod_regexp.dll
LIB_EXT=dll
RM=cmd /c del
LDFLAG=
else
EXE=extension
EXT=sqlite3_mod_regexp.so
RM=rm
ifeq ($(shell uname -s),Darwin)
LIB_EXT=dylib
else
LIB_EXT=so
endif
RM=rm -f
LDFLAG=-fPIC
endif
LIB=sqlite3_mod_regexp.$(LIB_EXT)

all : $(EXE) $(EXT)
all : $(EXE) $(LIB)

$(EXE) : extension.go
go build $<

$(EXT) : sqlite3_mod_regexp.c
$(LIB) : sqlite3_mod_regexp.c
gcc $(LDFLAG) -shared -o $@ $< -lsqlite3 -lpcre

clean :
@-$(RM) $(EXE) $(EXT)
@-$(RM) $(EXE) $(LIB)
6 changes: 5 additions & 1 deletion _example/mod_regexp/sqlite3_mod_regexp.c
Expand Up @@ -13,9 +13,13 @@ static void regexp_func(sqlite3_context *context, int argc, sqlite3_value **argv
int vec[500];
int n, rc;
pcre* re = pcre_compile(pattern, 0, &errstr, &erroff, NULL);
if (!re) {
sqlite3_result_error(context, errstr, 0);
return;
}
rc = pcre_exec(re, NULL, target, strlen(target), 0, 0, vec, 500);
if (rc <= 0) {
sqlite3_result_error(context, errstr, 0);
sqlite3_result_int(context, 0);
return;
}
sqlite3_result_int(context, 1);
Expand Down
17 changes: 11 additions & 6 deletions _example/mod_vtable/Makefile
@@ -1,24 +1,29 @@
ifeq ($(OS),Windows_NT)
EXE=extension.exe
EXT=sqlite3_mod_vtable.dll
LIB_EXT=dll
RM=cmd /c del
LIBCURL=-lcurldll
LDFLAG=
else
EXE=extension
EXT=sqlite3_mod_vtable.so
RM=rm
ifeq ($(shell uname -s),Darwin)
LIB_EXT=dylib
else
LIB_EXT=so
endif
RM=rm -f
LDFLAG=-fPIC
LIBCURL=-lcurl
endif
LIB=sqlite3_mod_vtable.$(LIB_EXT)

all : $(EXE) $(EXT)
all : $(EXE) $(LIB)

$(EXE) : extension.go
go build $<

$(EXT) : sqlite3_mod_vtable.cc
$(LIB) : sqlite3_mod_vtable.cc
g++ $(LDFLAG) -shared -o $@ $< -lsqlite3 $(LIBCURL)

clean :
@-$(RM) $(EXE) $(EXT)
@-$(RM) $(EXE) $(LIB)
2 changes: 1 addition & 1 deletion _example/mod_vtable/sqlite3_mod_vtable.cc
@@ -1,6 +1,6 @@
#include <string>
#include <sstream>
#include <sqlite3-binding.h>
#include <sqlite3.h>
#include <sqlite3ext.h>
#include <curl/curl.h>
#include "picojson.h"
Expand Down
8 changes: 4 additions & 4 deletions backup.go
@@ -1,4 +1,4 @@
// Copyright (C) 2014 Yasuhiro Matsumoto <mattn.jp@gmail.com>.
// Copyright (C) 2019 Yasuhiro Matsumoto <mattn.jp@gmail.com>.
//
// Use of this source code is governed by an MIT-style
// license that can be found in the LICENSE file.
Expand All @@ -25,18 +25,18 @@ type SQLiteBackup struct {
}

// Backup make backup from src to dest.
func (c *SQLiteConn) Backup(dest string, conn *SQLiteConn, src string) (*SQLiteBackup, error) {
func (destConn *SQLiteConn) Backup(dest string, srcConn *SQLiteConn, src string) (*SQLiteBackup, error) {
destptr := C.CString(dest)
defer C.free(unsafe.Pointer(destptr))
srcptr := C.CString(src)
defer C.free(unsafe.Pointer(srcptr))

if b := C.sqlite3_backup_init(c.db, destptr, conn.db, srcptr); b != nil {
if b := C.sqlite3_backup_init(destConn.db, destptr, srcConn.db, srcptr); b != nil {
bb := &SQLiteBackup{b: b}
runtime.SetFinalizer(bb, (*SQLiteBackup).Finish)
return bb, nil
}
return nil, c.lastError()
return nil, destConn.lastError()
}

// Step to backs up for one step. Calls the underlying `sqlite3_backup_step`
Expand Down

0 comments on commit 40fb371

Please sign in to comment.