Skip to content

Commit

Permalink
feature: mongodb driver distributed transaction
Browse files Browse the repository at this point in the history
  • Loading branch information
@rentiansheng
rentiansheng committed Jan 13, 2021
1 parent 6e2c3a3 commit 314f4f3
Show file tree
Hide file tree
Showing 7 changed files with 814 additions and 159 deletions.
198 changes: 39 additions & 159 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,204 +1,84 @@
<p align="center"><img src="etc/assets/mongo-gopher.png" width="250"></p>
<p align="center">
<a href="https://goreportcard.com/report/go.mongodb.org/mongo-driver"><img src="https://goreportcard.com/badge/go.mongodb.org/mongo-driver"></a>
<a href="https://godoc.org/go.mongodb.org/mongo-driver/mongo"><img src="etc/assets/godoc-mongo-blue.svg" alt="GoDoc"></a>
<a href="https://godoc.org/go.mongodb.org/mongo-driver/bson"><img src="etc/assets/godoc-bson-blue.svg" alt="GoDoc"></a>
<a href="https://docs.mongodb.com/ecosystem/drivers/go/"><img src="etc/assets/docs-mongodb-green.svg"></a>
</p>
[中文文档](README_zh-cn.md)

# MongoDB Go Driver
## Use mongodb driver to solve distributed transactions

The MongoDB supported driver for Go.

-------------------------
- [Requirements](#requirements)
- [Installation](#installation)
- [Usage](#usage)
- [Bugs / Feature Reporting](#bugs--feature-reporting)
- [Testing / Development](#testing--development)
- [Continuous Integration](#continuous-integration)
- [License](#license)
#### Solving distributed transaction scenarios

-------------------------
## Requirements
This project can only solve distributed transactions in multiple tables in the same database in the same replica-set in mongodb.

- Go 1.10 or higher. We aim to support the latest supported versions of go.
- MongoDB 2.6 and higher.

-------------------------
## Installation

The recommended way to get started using the MongoDB Go driver is by using `dep` to install the dependency in your project.
#### Main principles

```bash
dep ensure -add "go.mongodb.org/mongo-driver/mongo@~1.1.2"
```

-------------------------
## Usage
The mongodb server does not strictly bind the transaction in which the transaction is in, and the mongodb driver can initiate transactions to achieve this.
Changing this solution will extend the mongodb golang driver code,
It can be used for verification in mongodb driver 1.1.2 release and mongodb 4.0.x-4.2.x replic-set mode

To get started with the driver, import the `mongo` package, create a `mongo.Client`:
###### mongodb golang driver code extension content

```go
import (
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)

client, err := mongo.NewClient(options.Client().ApplyURI("mongodb://localhost:27017"))
Extension code:
```

And connect it to your running MongoDB server:

```go
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
mongo/session_exposer.go
x/mongo/driver/session/session_ext.go
```
The extension code is mainly the underlying logic, used to activate the transaction and bind the transaction. Does not contain business logic

To do this in a single step, you can use the `Connect` function:

```go
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
```

Calling `Connect` does not block for server discovery. If you wish to know if a MongoDB server has been found and connected to,
use the `Ping` method:
Related test cases:

```go
ctx, _ = context.WithTimeout(context.Background(), 2*time.Second)
err = client.Ping(ctx, readpref.Primary())
```
Transaction/Transaction_test.go
To insert a document into a collection, first retrieve a `Database` and then `Collection` instance from the `Client`:

```go
collection := client.Database("testing").Collection("numbers")
```

The `Collection` instance can then be used to insert documents:

```go
ctx, _ = context.WithTimeout(context.Background(), 5*time.Second)
res, err := collection.InsertOne(ctx, bson.M{"name": "pi", "value": 3.14159})
id := res.InsertedID
```

Several query methods return a cursor, which can be used like this:

```go
ctx, _ = context.WithTimeout(context.Background(), 30*time.Second)
cur, err := collection.Find(ctx, bson.D{})
if err != nil { log.Fatal(err) }
defer cur.Close(ctx)
for cur.Next(ctx) {
var result bson.M
err := cur.Decode(&result)
if err != nil { log.Fatal(err) }
// do something with result....
}
if err := cur.Err(); err != nil {
log.Fatal(err)
}
```

For methods that return a single item, a `SingleResult` instance is returned:

```go
var result struct {
Value float64
}
filter := bson.M{"name": "pi"}
ctx, _ = context.WithTimeout(context.Background(), 5*time.Second)
err = collection.FindOne(ctx, filter).Decode(&result)
if err != nil {
log.Fatal(err)
}
// Do something with result...
```

Additional examples and documentation can be found under the examples directory and [on the MongoDB Documentation website](https://docs.mongodb.com/ecosystem/drivers/go/).

-------------------------
## Bugs / Feature Reporting

New Features and bugs can be reported on jira: https://jira.mongodb.org/browse/GODRIVER

-------------------------
## Testing / Development

The driver tests can be run against several database configurations. The most simple configuration is a standalone mongod with no auth, no ssl, and no compression. To run these basic driver tests, make sure a standalone MongoDB server instance is running at localhost:27017. To run the tests, you can run `make` (on Windows, run `nmake`) with the following:
Upper-level use logic:

```
TOPOLOGY=server make
Transaction/Transaction.go
```

The `TOPOLOGY`variable must be set to run tests. This will run coverage, run go-lint, run go-vet, and build the examples.
Encapsulate business logic, realize business-level management interfaces such as opening, committing, and rolling back transactions, and provide an operation interface for the transaction uuid and the cursor id record operation of the statement execution within the transaction.
In actual use, the transaction uuid and in-service statement execution cursor id need to be stored centrally, and all service instances need to be available and used. You can use redis and mysql as central storage

### Testing Different Topologies

To test a **replica set**, set `MONGODB_URI="<connection-string>"` and `TOPOLOGY=replica_set` for the `make` command. For example, for a local replica set named `rs1` comprised of three nodes on ports 27017, 27018, and 27019:
###### Specific implementation

```
MONGODB_URI="mongodb://localhost:27017,localhost:27018,localhost:27018/?replicaSet=rs1" TOPOLOGY=replica_set make
```
1. Activate a transaction, generate a transaction uuid (mongodb driver provides the generation method), Transaction/Transaction.go: StartTransaction
2. Activate a session through the uuid of the transaction and join a transaction of the mongodb server, Transaction/Transaction.go: ReloadSession
3. Bind the transaction to the session, get the SessionContext, mongo/session_exposer.go:TxnContextWithSession
4. Transaction execution needs to move the cursor, Transaction/Transaction.go: NextTransactionCursor
5. Perform curl operations

To test a **sharded cluster**, set `MONGODB_URI="<connection-string>"` and `TOPOLOGY=sharded_cluster` variables for the `make` command. For example, for a sharded cluster with a single mongos on port 27017:

```
MONGODB_URI="mongodb://localhost:27017/" TOPOLOGY=sharder_cluster make
```

### Testing Auth and SSL
#### requires attention

To test authentication and SSL, first set up a MongoDB cluster with auth and SSL configured. Testing authentication requires a user with the `root` role on the `admin` database. The Go Driver repository comes with example certificates in the `data/certificates` directory. These certs can be used for testing. Here is an example command that would run a mongod with SSL correctly configured for tests:
-The TxnNumber in the current code is only for testing and cannot be used in online multi-service scenarios
-Uuid in TxnNumber must use go.mongodb.org/mongo-driver/x/mongo/driver/uuid base64 encoded string value

```
mongod \
--auth \
--sslMode requireSSL \
--sslPEMKeyFile $(pwd)/data/certificates/server.pem \
--sslCAFile $(pwd)/data/certificates/ca.pem \
--sslWeakCertificateValidation
```

To run the tests with `make`, set `MONGO_GO_DRIVER_CA_FILE` to the location of the CA file used by the database, set `MONGODB_URI` to the connection string of the server, set `AUTH=auth`, and set `SSL=ssl`. For example:

```
AUTH=auth SSL=ssl MONGO_GO_DRIVER_CA_FILE=$(pwd)/data/certificates/ca.pem MONGODB_URI="mongodb://user:password@localhost:27017/?authSource=admin" make
```

Notes:
- The `--sslWeakCertificateValidation` flag is required on the server for the test suite to work correctly.
- The test suite requires the auth database to be set with `?authSource=admin`, not `/admin`.

### Testing Compression

The MongoDB Go Driver supports wire protocol compression using Snappy or zLib. To run tests with wire protocol compression, set `MONGO_GO_DRIVER_COMPRESSOR` to `snappy` or `zlib`. For example:

```
MONGO_GO_DRIVER_COMPRESSOR=snappy make
```
#### Solution discovery and implementation

Ensure the [`--networkMessageCompressors` flag](https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-networkmessagecompressors) on mongod or mongos includes `zlib` if testing zLib compression.
participants:
[rentiansheng](https://github.com/rentiansheng)

-------------------------
## Feedback
[wusendong](https://github.com/wusendong)

The MongoDB Go Driver is not feature complete, so any help is appreciated. Check out the [project page](https://jira.mongodb.org/browse/GODRIVER)
for tickets that need completing. See our [contribution guidelines](CONTRIBUTING.md) for details.
[breezelxp](https://github.com/breezelxp)

-------------------------
## Continuous Integration

Commits to master are run automatically on [evergreen](https://evergreen.mongodb.com/waterfall/mongo-go-driver).
Use project:

-------------------------
## Thanks and Acknowledgement

<a href="https://github.com/ashleymcnamara">@ashleymcnamara</a> - Mongo Gopher Artwork
[Blue Whale Configuration Platform](https://github.com/Tencent/bk-cmdb)

-------------------------
## License
#### Drainage
[bson decode register](https://github.com/rentiansheng/bson-register)

The MongoDB Go Driver is licensed under the [Apache License](LICENSE).
Bson decoding interface priority uses the actual type as the final object, such as slice, map
In the mongodb golang official driver, when bson uses interface at the top level, it puts slice and map in an object called primitive.D (golang []interface type).

0 comments on commit 314f4f3

Please sign in to comment.