Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize mongodb plugin #7207

Merged
merged 8 commits into from May 6, 2019
Merged

Optimize mongodb plugin #7207

merged 8 commits into from May 6, 2019

Conversation

@UMU618
Copy link
Contributor

UMU618 commented Apr 25, 2019

mongodb plugin performance greatly enhanced

before:
fc::variant -> JSON string -> bson

now:
fc::variant -> bson

the conversion performance is 40~50x than before, when processes big blocks.

Just add bson.hpp, hope it will pass all checks and tests.

UMU618 added 3 commits Mar 25, 2019
Improve for MongoDB sharding
@UMU618

This comment has been minimized.

Copy link
Contributor Author

UMU618 commented Apr 25, 2019

2 failing and 15 successful checks
I just add bson.hpp, not even modify cpp, and the checks fail?

@UMU618

This comment has been minimized.

Copy link
Contributor Author

UMU618 commented Apr 25, 2019

mongo_db_plugin.cpp modified

convert fc::variant to BSON directly

@heifner

This comment has been minimized.

Copy link
Contributor

heifner commented Apr 25, 2019

The nodeos_run_test-mongodb test is failing. It looks like because the format that the python expects for NumberLong has changed. Can you look at updating the test?

@heifner

This comment has been minimized.

Copy link
Contributor

heifner commented Apr 25, 2019

    cmd returned transaction id: 687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08, status: executed, (possible) block num: 48 

          cmd: echo 'db.transactions.findOne( { "trx_id": "687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08" } )' | mongo --host localhost --port 27017 EOStest

           ERROR: JSONDecodeError

           Raw MongoDB response: > MongoDB shell version v3.6.3

connecting to: mongodb://localhost:27017/EOStest

MongoDB server version: 3.6.3

{

	"_id" : ObjectId("5cc1646325fb28b2a8b31761"),

	"trx_id" : "687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08",

	"accepted" : true,

	"actions" : [

		{

			"account" : "eosio",

			"name" : "newaccount",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"creator" : "defproducera",

				"name" : "currency1111",

				"owner" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS72VGHvEb8Rwh3tYvNei3NeMRMewPJot3UXojHSv1PUDAiQdvzZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				},

				"active" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS7RkUJbfvyhWPJSFQrPpiCgX5SFvi9DWdssZn2Hn8gVE3K1WzVZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				}

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4601000000010003198dbddbf5f1afe834488c2c60291156860cf4ab3344ee9ee01c75c4fa607f5201000000010000000100034e5f928558c398ecfef966c2e9b0ca3e27ed088cf73dff8419ad7a38651ed44c01000000"

		},

		{

			"account" : "eosio",

			"name" : "buyram",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"payer" : "defproducera",

				"receiver" : "currency1111",

				"quant" : "200000.0000 SYS"

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4600943577000000000453595300000000"

		},

		{

			"account" : "eosio",

			"name" : "delegatebw",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"from" : "defproducera",

				"receiver" : "currency1111",

				"stake_net_quantity" : "100.0000 SYS",

				"stake_cpu_quantity" : "100.0000 SYS",

				"transfer" : NumberLong(0)

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4640420f0000000000045359530000000040420f0000000000045359530000000000"

		}

	],

	"context_free_actions" : [ ],

	"context_free_data" : [ ],

	"createdAt" : ISODate("2019-04-25T07:40:19.933Z"),

	"delay_sec" : NumberLong(0),

	"expiration" : "2019-04-25T07:40:49",

	"implicit" : false,

	"max_cpu_usage_ms" : NumberLong(0),

	"max_net_usage_words" : NumberLong(0),

	"ref_block_num" : NumberLong(30),

	"ref_block_prefix" : NumberLong(939910211),

	"scheduled" : false,

	"signatures" : [

		"SIG_K1_KgdrasgxWZHQSMyUVPfQGogfConEQppufWVb3oUSkbL5EJXYkVFNRvmwchPBAiAVU48sKnmMoPwP4V4XN74XjZmWDz2TcY"

	],

	"signing_keys" : [

		"EOS6mV8wVdsLpvh7M1TfGK4cQapt8HZMtgXEA5MuK9EyGZfv4cPu1"

	],

	"transaction_extensions" : [ ]

}

bye



           Normalized MongoDB response: > {

	"_id" : "ObjectId-5cc1646325fb28b2a8b31761",

	"trx_id" : "687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08",

	"accepted" : true,

	"actions" : [

		{

			"account" : "eosio",

			"name" : "newaccount",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"creator" : "defproducera",

				"name" : "currency1111",

				"owner" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS72VGHvEb8Rwh3tYvNei3NeMRMewPJot3UXojHSv1PUDAiQdvzZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				},

				"active" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS7RkUJbfvyhWPJSFQrPpiCgX5SFvi9DWdssZn2Hn8gVE3K1WzVZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				}

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4601000000010003198dbddbf5f1afe834488c2c60291156860cf4ab3344ee9ee01c75c4fa607f5201000000010000000100034e5f928558c398ecfef966c2e9b0ca3e27ed088cf73dff8419ad7a38651ed44c01000000"

		},

		{

			"account" : "eosio",

			"name" : "buyram",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"payer" : "defproducera",

				"receiver" : "currency1111",

				"quant" : "200000.0000 SYS"

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4600943577000000000453595300000000"

		},

		{

			"account" : "eosio",

			"name" : "delegatebw",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"from" : "defproducera",

				"receiver" : "currency1111",

				"stake_net_quantity" : "100.0000 SYS",

				"stake_cpu_quantity" : "100.0000 SYS",

				"transfer" : NumberLong(0)

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4640420f0000000000045359530000000040420f0000000000045359530000000000"

		}

	],

	"context_free_actions" : [ ],

	"context_free_data" : [ ],

	"createdAt" : "ISODate-2019-04-25T07:40:19.933Z",

	"delay_sec" : NumberLong(0),

	"expiration" : "2019-04-25T07:40:49",

	"implicit" : false,

	"max_cpu_usage_ms" : NumberLong(0),

	"max_net_usage_words" : NumberLong(0),

	"ref_block_num" : NumberLong(30),

	"ref_block_prefix" : NumberLong(939910211),

	"scheduled" : false,

	"signatures" : [

		"SIG_K1_KgdrasgxWZHQSMyUVPfQGogfConEQppufWVb3oUSkbL5EJXYkVFNRvmwchPBAiAVU48sKnmMoPwP4V4XN74XjZmWDz2TcY"

	],

	"signing_keys" : [

		"EOS6mV8wVdsLpvh7M1TfGK4cQapt8HZMtgXEA5MuK9EyGZfv4cPu1"

	],

	"transaction_extensions" : [ ]

}

 Test failed.
@heifner

This comment has been minimized.

Copy link
Contributor

heifner commented Apr 25, 2019

2 failing and 15 successful checks
I just add bson.hpp, not even modify cpp, and the checks fail?

You likely caught develop in a bad state. What I posted above seems to be the only issue.

@UMU618

This comment has been minimized.

Copy link
Contributor Author

UMU618 commented Apr 26, 2019

2 failing and 15 successful checks
I just add bson.hpp, not even modify cpp, and the checks fail?

You likely caught develop in a bad state. What I posted above seems to be the only issue.

Fixed. All checks have passed @heifner

UMU618 and others added 3 commits May 6, 2019
@heifner
heifner approved these changes May 6, 2019
@heifner heifner merged commit d458d72 into EOSIO:develop May 6, 2019
24 checks passed
24 checks passed
buildkite/eosio Build #11581 passed (42 minutes, 50 seconds)
Details
buildkite/eosio/aws-amazon-linux-2-build Passed (9 minutes, 18 seconds)
Details
buildkite/eosio/aws-amazon-linux-2-np-tests Passed (13 minutes, 4 seconds)
Details
buildkite/eosio/aws-amazon-linux-2-tests Passed (3 minutes)
Details
buildkite/eosio/bar-chart-test-metrics Passed (28 seconds)
Details
buildkite/eosio/centos-centos-7-build Passed (11 minutes, 20 seconds)
Details
buildkite/eosio/centos-centos-7-np-tests Passed (13 minutes, 40 seconds)
Details
buildkite/eosio/centos-centos-7-package-builder Passed (2 minutes, 34 seconds)
Details
buildkite/eosio/centos-centos-7-tests Passed (4 minutes, 7 seconds)
Details
buildkite/eosio/darwin-brew-updater Passed (9 seconds)
Details
buildkite/eosio/darwin-mojave-build Passed (8 minutes, 22 seconds)
Details
buildkite/eosio/darwin-mojave-np-tests Passed (24 minutes, 59 seconds)
Details
buildkite/eosio/darwin-mojave-package-builder Passed (33 seconds)
Details
buildkite/eosio/darwin-mojave-tests Passed (2 minutes, 38 seconds)
Details
buildkite/eosio/git-submodule-regression-check Passed (12 seconds)
Details
buildkite/eosio/pipeline Passed (22 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-16-dot-04-build Passed (9 minutes, 32 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-16-dot-04-np-tests Passed (12 minutes, 59 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-16-dot-04-package-builder Passed (2 minutes, 57 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-16-dot-04-tests Passed (3 minutes, 34 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-18-dot-04-build Passed (10 minutes, 20 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-18-dot-04-np-tests Passed (11 minutes, 31 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-18-dot-04-package-builder Passed (2 minutes, 15 seconds)
Details
buildkite/eosio/ubuntu-ubuntu-18-dot-04-tests Passed (3 minutes, 29 seconds)
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.