Skip to content

Optimize the API to get transactions #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 10, 2019
Merged

Optimize the API to get transactions #253

merged 1 commit into from
May 10, 2019

Conversation

@sgkim126
Copy link
Contributor

@sgkim126 sgkim126 commented May 8, 2019

The previous query joins all rows of tables to calculate offset even though it needs only itemsPerPage rows.

This patch splits the query to get transactions. The first query gets the transaction hash that the second query selects. It helps the second query not to scan all the table of the rows of included tables.

@sgkim126 sgkim126 added the optimization This makes the system faster label May 8, 2019
@sgkim126 sgkim126 requested a review from joojis May 8, 2019 05:10
@sgkim126
Copy link
Contributor Author

sgkim126 commented May 8, 2019 

curl -X GET "http://localhost:9001/api/tx" -H "accept: application/json"

The current implementation uses the below query for the above API.

SELECT "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."blockHash", "Transaction"."tracker", "Transaction"."transactionIndex", "Transaction"."type", "Transaction"."seq", "Transaction"."fee", "Transaction"."networkId", "Transaction"."sig",
    "Transaction"."signer", "Transaction"."errorHint", "Transaction"."timestamp", "Transaction"."isPending", "Transaction"."pendingTimestamp", "Transaction"."createdAt", "Transaction"."updatedAt", "mintAsset"."transactionHash" AS "mintAsset.transactionHash",
    "mintAsset"."networkId" AS "mintAsset.networkId", "mintAsset"."shardId" AS "mintAsset.shardId", "mintAsset"."metadata" AS "mintAsset.metadata", "mintAsset"."approver" AS "mintAsset.approver", "mintAsset"."registrar" AS "mintAsset.registrar",
    "mintAsset"."allowedScriptHashes" AS "mintAsset.allowedScriptHashes", "mintAsset"."approvals" AS "mintAsset.approvals", "mintAsset"."lockScriptHash" AS "mintAsset.lockScriptHash", "mintAsset"."parameters" AS "mintAsset.parameters",
    "mintAsset"."supply" AS "mintAsset.supply", "mintAsset"."assetName" AS "mintAsset.assetName", "mintAsset"."assetType" AS "mintAsset.assetType", "mintAsset"."recipient" AS "mintAsset.recipient", "mintAsset"."createdAt" AS "mintAsset.createdAt",
    "mintAsset"."updatedAt" AS "mintAsset.updatedAt", "transferAsset"."transactionHash" AS "transferAsset.transactionHash", "transferAsset"."networkId" AS "transferAsset.networkId", "transferAsset"."metadata" AS "transferAsset.metadata",
    "transferAsset"."approvals" AS "transferAsset.approvals", "transferAsset"."expiration" AS "transferAsset.expiration", "transferAsset"."inputs" AS "transferAsset.inputs", "transferAsset"."burns" AS "transferAsset.burns",
    "transferAsset"."outputs" AS "transferAsset.outputs", "transferAsset"."orders" AS "transferAsset.orders", "transferAsset"."createdAt" AS "transferAsset.createdAt", "transferAsset"."updatedAt" AS "transferAsset.updatedAt",
    "composeAsset"."transactionHash" AS "composeAsset.transactionHash", "composeAsset"."networkId" AS "composeAsset.networkId", "composeAsset"."shardId" AS "composeAsset.shardId", "composeAsset"."metadata" AS "composeAsset.metadata",
    "composeAsset"."approver" AS "composeAsset.approver", "composeAsset"."registrar" AS "composeAsset.registrar", "composeAsset"."allowedScriptHashes" AS "composeAsset.allowedScriptHashes", "composeAsset"."approvals" AS "composeAsset.approvals",
    "composeAsset"."lockScriptHash" AS "composeAsset.lockScriptHash", "composeAsset"."parameters" AS "composeAsset.parameters", "composeAsset"."supply" AS "composeAsset.supply", "composeAsset"."assetName" AS "composeAsset.assetName",
    "composeAsset"."assetType" AS "composeAsset.assetType", "composeAsset"."recipient" AS "composeAsset.recipient", "composeAsset"."inputs" AS "composeAsset.inputs", "composeAsset"."createdAt" AS "composeAsset.createdAt",
    "composeAsset"."updatedAt" AS "composeAsset.updatedAt", "decomposeAsset"."transactionHash" AS "decomposeAsset.transactionHash", "decomposeAsset"."networkId" AS "decomposeAsset.networkId", "decomposeAsset"."approvals" AS "decomposeAsset.approvals",
    "decomposeAsset"."input" AS "decomposeAsset.input", "decomposeAsset"."outputs" AS "decomposeAsset.outputs", "decomposeAsset"."createdAt" AS "decomposeAsset.createdAt", "decomposeAsset"."updatedAt" AS "decomposeAsset.updatedAt",
    "changeAssetScheme"."transactionHash" AS "changeAssetScheme.transactionHash", "changeAssetScheme"."assetType" AS "changeAssetScheme.assetType", "changeAssetScheme"."networkId" AS "changeAssetScheme.networkId",
    "changeAssetScheme"."shardId" AS "changeAssetScheme.shardId", "changeAssetScheme"."metadata" AS "changeAssetScheme.metadata", "changeAssetScheme"."approver" AS "changeAssetScheme.approver", "changeAssetScheme"."registrar" AS "changeAssetScheme.registrar",
    "changeAssetScheme"."allowedScriptHashes" AS "changeAssetScheme.allowedScriptHashes", "changeAssetScheme"."approvals" AS "changeAssetScheme.approvals", "changeAssetScheme"."seq" AS "changeAssetScheme.seq",
    "changeAssetScheme"."createdAt" AS  "changeAssetScheme.createdAt", "changeAssetScheme"."updatedAt" AS "changeAssetScheme.updatedAt", "increaseAssetSupply"."transactionHash" AS "increaseAssetSupply.transactionHash",
    "increaseAssetSupply"."networkId" AS "increaseAssetSupply.networkId", "increaseAssetSupply"."shardId" AS "increaseAssetSupply.shardId", "increaseAssetSupply"."assetType" AS "increaseAssetSupply.assetType",
    "increaseAssetSupply"."approvals" AS "increaseAssetSupply.approvals", "increaseAssetSupply"."seq" AS "increaseAssetSupply.seq", "increaseAssetSupply"."lockScriptHash" AS "increaseAssetSupply.lockScriptHash",
    "increaseAssetSupply"."parameters" AS "increaseAssetSupply.parameters", "increaseAssetSupply"."supply" AS "increaseAssetSupply.supply", "increaseAssetSupply"."recipient" AS "increaseAssetSupply.recipient",
    "increaseAssetSupply"."createdAt" AS "increaseAssetSupply.createdAt", "increaseAssetSupply"."updatedAt" AS "increaseAssetSupply.updatedAt", "wrapCCC"."transactionHash" AS "wrapCCC.transactionHash",
    "wrapCCC"."shardId" AS "wrapCCC.shardId", "wrapCCC"."lockScriptHash" AS "wrapCCC.lockScriptHash", "wrapCCC"."parameters" AS "wrapCCC.parameters", "wrapCCC"."quantity" AS "wrapCCC.quantity", "wrapCCC"."recipient" AS "wrapCCC.recipient",
    "wrapCCC"."createdAt" AS "wrapCCC.createdAt", "wrapCCC"."updatedAt" AS "wrapCCC.updatedAt", "unwrapCCC"."transactionHash" AS "unwrapCCC.transactionHash", "unwrapCCC"."receiver" AS "unwrapCCC.receiver", "unwrapCCC"."burn" AS "unwrapCCC.burn",
    "unwrapCCC"."createdAt" AS "unwrapCCC.createdAt", "unwrapCCC"."updatedAt" AS "unwrapCCC.updatedAt", "pay"."transactionHash" AS "pay.transactionHash", "pay"."receiver" AS "pay.receiver", "pay"."quantity" AS "pay.quantity", "pay"."createdAt" AS "pay.createdAt",
    "pay"."updatedAt" AS "pay.updatedAt", "setRegularKey"."transactionHash" AS "setRegularKey.transactionHash", "setRegularKey"."key" AS "setRegularKey.key", "setRegularKey"."createdAt" AS "setRegularKey.createdAt", "setRegularKey"."updatedAt" AS "setRegularKey.updatedAt",
    "createShard"."transactionHash" AS "createShard.transactionHash", "createShard"."shardId" AS "createShard.shardId", "createShard"."users" AS "createShard.users", "createShard"."createdAt" AS "createShard.createdAt", "createShard"."updatedAt" AS "createShard.updatedAt",
    "setShardOwners"."transactionHash" AS "setShardOwners.transactionHash", "setShardOwners"."shardId" AS "setShardOwners.shardId", "setShardOwners"."owners" AS "setShardOwners.owners", "setShardOwners"."createdAt" AS "setShardOwners.createdAt",
    "setShardOwners"."updatedAt" AS "setShardOwners.updatedAt", "setShardUsers"."transactionHash" AS "setShardUsers.transactionHash", "setShardUsers"."shardId" AS "setShardUsers.shardId", "setShardUsers"."users" AS "setShardUsers.users",
    "setShardUsers"."createdAt" AS "setShardUsers.createdAt", "setShardUsers"."updatedAt" AS "setShardUsers.updatedAt", "store"."transactionHash" AS "store.transactionHash", "store"."content" AS "store.content", "store"."certifier" AS "store.certifier",
    "store"."signature" AS "store.signature", "store"."createdAt" AS "store.createdAt", "store"."updatedAt" AS "store.updatedAt", "remove"."transactionHash" AS "remove.transactionHash", "remove"."textHash" AS "remove.textHash", "remove"."signature" AS "remove.signature",
    "remove"."createdAt" AS "remove.createdAt", "remove"."updatedAt" AS "remove.updatedAt", "custom"."transactionHash" AS "custom.transactionHash", "custom"."handlerId" AS "custom.handlerId", "custom"."content" AS "custom.content", "custom"."createdAt" AS "custom.createdAt", "custom"."updatedAt" AS "custom.updatedAt"
FROM "Transactions" AS "Transaction"
LEFT OUTER JOIN "MintAssets" AS "mintAsset" ON "Transaction"."hash" = "mintAsset"."transactionHash"
LEFT OUTER JOIN "TransferAssets" AS "transferAsset" ON "Transaction"."hash" = "transferAsset"."transactionHash"
LEFT OUTER JOIN "ComposeAssets" AS "composeAsset" ON "Transaction"."hash" = "composeAsset"."transactionHash"
LEFT OUTER JOIN "DecomposeAssets" AS "decomposeAsset" ON "Transaction"."hash" = "decomposeAsset"."transactionHash"
LEFT OUTER JOIN "ChangeAssetSchemes" AS "changeAssetScheme" ON "Transaction"."hash" = "changeAssetScheme"."transactionHash"
LEFT OUTER JOIN "IncreaseAssetSupplies" AS "increaseAssetSupply" ON "Transaction"."hash" = "increaseAssetSupply"."transactionHash"
LEFT OUTER JOIN "WrapCCCs" AS "wrapCCC" ON "Transaction"."hash" = "wrapCCC"."transactionHash"
LEFT OUTER JOIN "UnwrapCCCs" AS "unwrapCCC" ON "Transaction"."hash" = "unwrapCCC"."transactionHash"
LEFT OUTER JOIN "Pays" AS "pay" ON "Transaction"."hash" = "pay"."transactionHash"
LEFT OUTER JOIN "SetRegularKeys" AS "setRegularKey" ON "Transaction"."hash" = "setRegularKey"."transactionHash"
LEFT OUTER JOIN "CreateShards" AS "createShard" ON "Transaction"."hash" = "createShard"."transactionHash"
LEFT OUTER JOIN "SetShardOwners" AS "setShardOwners" ON "Transaction"."hash" = "setShardOwners"."transactionHash"
LEFT OUTER JOIN "SetShardUsers" AS "setShardUsers" ON "Transaction"."hash" = "setShardUsers"."transactionHash"
LEFT OUTER JOIN "Stores" AS "store" ON "Transaction"."hash" = "store"."transactionHash"
LEFT OUTER JOIN "Removes" AS "remove" ON "Transaction"."hash" = "remove"."transactionHash"
LEFT OUTER JOIN "Customs" AS "custom" ON "Transaction"."hash" = "custom"."transactionHash"
WHERE "Transaction"."isPending" = false
ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
LIMIT 15 OFFSET 0;

The query plan looks like the below.

Limit  (cost=421008.58..421008.61 rows=15 width=17217)
   ->  Sort  (cost=421008.58..422412.78 rows=561681 width=17217)
         Sort Key: "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
         ->  Merge Left Join  (cost=4.59..407228.04 rows=561681 width=17217)
               Merge Cond: (("Transaction".hash)::text = (custom."transactionHash")::text)
               ->  Merge Left Join  (cost=4.45..405773.77 rows=561681 width=16165)
                     Merge Cond: (("Transaction".hash)::text = (remove."transactionHash")::text)
                     ->  Merge Left Join  (cost=4.31..404324.05 rows=561681 width=14601)
                           Merge Cond: (("Transaction".hash)::text = (store."transactionHash")::text)
                           ->  Merge Left Join  (cost=4.17..402874.89 rows=561681 width=12521)
                                 Merge Cond: (("Transaction".hash)::text = ("setShardUsers"."transactionHash")::text)
                                 ->  Merge Left Join  (cost=4.03..401422.96 rows=561681 width=11953)
                                       Merge Cond: (("Transaction".hash)::text = ("setShardOwners"."transactionHash")::text)
                                       ->  Merge Left Join  (cost=3.88..399971.04 rows=561681 width=11385)
                                             Merge Cond: (("Transaction".hash)::text = ("createShard"."transactionHash")::text)
                                             ->  Merge Left Join  (cost=3.74..398519.12 rows=561681 width=10817)
                                                   Merge Cond: (("Transaction".hash)::text = ("setRegularKey"."transactionHash")::text)
                                                   ->  Merge Left Join  (cost=3.46..396724.23 rows=561681 width=10604)
                                                         Merge Cond: (("Transaction".hash)::text = (pay."transactionHash")::text)
                                                         ->  Merge Left Join  (cost=3.04..334559.88 rows=561681 width=10474)
                                                               Merge Cond: (("Transaction".hash)::text = ("unwrapCCC"."transactionHash")::text)
                                                               ->  Merge Left Join  (cost=2.63..323673.24 rows=561681 width=9424)
                                                                     Merge Cond: (("Transaction".hash)::text = ("wrapCCC"."transactionHash")::text)
                                                                     ->  Merge Left Join  (cost=2.22..319019.84 rows=561681 width=9198)
                                                                           Merge Cond: (("Transaction".hash)::text = ("increaseAssetSupply"."transactionHash")::text)
                                                                           ->  Merge Left Join  (cost=1.94..316605.52 rows=561681 width=8850)
                                                                                 Merge Cond: (("Transaction".hash)::text = ("changeAssetScheme"."transactionHash")::text)
                                                                                 ->  Merge Left Join  (cost=1.66..314083.05 rows=561681 width=8056)
                                                                                       Merge Cond: (("Transaction".hash)::text = ("decomposeAsset"."transactionHash")::text)
                                                                                       ->  Merge Left Join  (cost=1.52..312633.06 rows=561681 width=6912)
                                                                                             Merge Cond: (("Transaction".hash)::text = ("composeAsset"."transactionHash")::text)
                                                                                             ->  Merge Left Join  (cost=1.38..311196.44 rows=561681 width=2584)
                                                                                                   Merge Cond: (("Transaction".hash)::text = ("transferAsset"."transactionHash")::text)
                                                                                                   ->  Merge Left Join  (cost=0.96..169952.66 rows=561681 width=1009)
                                                                                                         Merge Cond: (("Transaction".hash)::text = ("mintAsset"."transactionHash")::text)
                                                                                                         ->  Index Scan using "Transactions_pkey" on "Transactions" "Transaction"  (cost=0.55..163737.76 rows=561681 width=458)
                                                                                                               Filter: (NOT "isPending")
                                                                                                         ->  Index Scan using "MintAssets_pkey" on "MintAssets" "mintAsset"  (cost=0.41..4635.11 rows=14047 width=551)
                                                                                                   ->  Index Scan using "TransferAssets_pkey" on "TransferAssets" "transferAsset"  (cost=0.42..137947.09 rows=151399 width=1575)
                                                                                             ->  Index Scan using "ComposeAssets_pkey" on "ComposeAssets" "composeAsset"  (cost=0.14..32.28 rows=10 width=4328)
                                                                                       ->  Index Scan using "DecomposeAssets_pkey" on "DecomposeAssets" "decomposeAsset"  (cost=0.14..45.04 rows=60 width=1144)
                                                                                 ->  Index Scan using "ChangeAssetSchemes_pkey" on "ChangeAssetSchemes" "changeAssetScheme"  (cost=0.28..1064.61 rows=4293 width=794)
                                                                           ->  Index Scan using "IncreaseAssetSupplies_pkey" on "IncreaseAssetSupplies" "increaseAssetSupply"  (cost=0.28..967.43 rows=3415 width=348)
                                                                     ->  Index Scan using "WrapCCCs_pkey" on "WrapCCCs" "wrapCCC"  (cost=0.41..3063.38 rows=14865 width=226)
                                                               ->  Index Scan using "UnwrapCCCs_pkey" on "UnwrapCCCs" "unwrapCCC"  (cost=0.41..9303.29 rows=14332 width=1050)
                                                         ->  Index Scan using "Pays_pkey" on "Pays" pay  (cost=0.42..56294.71 rows=357235 width=130)
                                                   ->  Index Scan using "SetRegularKeys_pkey" on "SetRegularKeys" "setRegularKey"  (cost=0.28..369.59 rows=1688 width=213)
                                             ->  Index Scan using "CreateShards_pkey" on "CreateShards" "createShard"  (cost=0.14..46.09 rows=130 width=568)
                                       ->  Index Scan using "SetShardOwners_pkey" on "SetShardOwners" "setShardOwners"  (cost=0.14..46.09 rows=130 width=568)
                                 ->  Index Scan using "SetShardUsers_pkey" on "SetShardUsers" "setShardUsers"  (cost=0.14..46.09 rows=130 width=568)
                           ->  Index Scan using "Stores_pkey" on "Stores" store  (cost=0.14..44.59 rows=30 width=2080)
                     ->  Index Scan using "Removes_pkey" on "Removes" remove  (cost=0.14..44.89 rows=50 width=1564)
               ->  Index Scan using "Customs_pkey" on "Customs" custom  (cost=0.14..49.19 rows=70 width=1052)

As you can see sorting(-> Sort (cost=421008.58..422412.78 rows=561681 width=17217) costs much.
This patch splits the above query into 2 queries.

SELECT "hash", "blockNumber", "transactionIndex"
FROM "Transactions" AS "Transaction"
WHERE "Transaction"."isPending" = false
ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
LIMIT 15 OFFSET 0;

SELECT "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."blockHash", "Transaction"."tracker", "Transaction"."transactionIndex", "Transaction"."type", "Transaction"."seq", "Transaction"."fee", "Transaction"."networkId",
    "Transaction"."sig", "Transaction"."signer", "Transaction"."errorHint", "Transaction"."timestamp", "Transaction"."isPending", "Transaction"."pendingTimestamp", "Transaction"."createdAt", "Transaction"."updatedAt", "mintAsset"."transactionHash" AS "mintAsset.transactionHash", "mintAsset"."networkId" AS "mintAsset.networkId", "mintAsset"."shardId" AS "mintAsset.shardId", "mintAsset"."metadata" AS "mintAsset.metadata", "mintAsset"."approver" AS "mintAsset.approver",
    "mintAsset"."registrar" AS "mintAsset.registrar", "mintAsset"."allowedScriptHashes" AS "mintAsset.allowedScriptHashes", "mintAsset"."approvals" AS "mintAsset.approvals", "mintAsset"."lockScriptHash" AS "mintAsset.lockScriptHash",
    "mintAsset"."parameters" AS "mintAsset.parameters", "mintAsset"."supply" AS "mintAsset.supply", "mintAsset"."assetName" AS "mintAsset.assetName", "mintAsset"."assetType" AS "mintAsset.assetType", "mintAsset"."recipient" AS "mintAsset.recipient",
    "mintAsset"."createdAt" AS "mintAsset.createdAt", "mintAsset"."updatedAt" AS "mintAsset.updatedAt", "transferAsset"."transactionHash" AS "transferAsset.transactionHash", "transferAsset"."networkId" AS "transferAsset.networkId",
    "transferAsset"."metadata" AS "transferAsset.metadata", "transferAsset"."approvals" AS "transferAsset.approvals", "transferAsset"."expiration" AS "transferAsset.expiration", "transferAsset"."inputs" AS "transferAsset.inputs",
    "transferAsset"."burns" AS "transferAsset.burns", "transferAsset"."outputs" AS "transferAsset.outputs", "transferAsset"."orders" AS "transferAsset.orders", "transferAsset"."createdAt" AS "transferAsset.createdAt",
    "transferAsset"."updatedAt" AS "transferAsset.updatedAt", "composeAsset"."transactionHash" AS "composeAsset.transactionHash", "composeAsset"."networkId" AS "composeAsset.networkId", "composeAsset"."shardId" AS "composeAsset.shardId",
    "composeAsset"."metadata" AS "composeAsset.metadata", "composeAsset"."approver" AS "composeAsset.approver", "composeAsset"."registrar" AS "composeAsset.registrar", "composeAsset"."allowedScriptHashes" AS "composeAsset.allowedScriptHashes",
    "composeAsset"."approvals" AS "composeAsset.approvals", "composeAsset"."lockScriptHash" AS "composeAsset.lockScriptHash", "composeAsset"."parameters" AS "composeAsset.parameters", "composeAsset"."supply" AS "composeAsset.supply",
    "composeAsset"."assetName" AS "composeAsset.assetName", "composeAsset"."assetType" AS "composeAsset.assetType", "composeAsset"."recipient" AS "composeAsset.recipient", "composeAsset"."inputs" AS "composeAsset.inputs",
    "composeAsset"."createdAt" AS "composeAsset.createdAt", "composeAsset"."updatedAt" AS "composeAsset.updatedAt", "decomposeAsset"."transactionHash" AS "decomposeAsset.transactionHash", "decomposeAsset"."networkId" AS "decomposeAsset.networkId",
    "decomposeAsset"."approvals" AS "decomposeAsset.approvals", "decomposeAsset"."input" AS "decomposeAsset.input", "decomposeAsset"."outputs" AS "decomposeAsset.outputs", "decomposeAsset"."createdAt" AS "decomposeAsset.createdAt",
    "decomposeAsset"."updatedAt" AS "decomposeAsset.updatedAt", "changeAssetScheme"."transactionHash" AS "changeAssetScheme.transactionHash", "changeAssetScheme"."assetType" AS "changeAssetScheme.assetType",
    "changeAssetScheme"."networkId" AS "changeAssetScheme.networkId", "changeAssetScheme"."shardId" AS "changeAssetScheme.shardId", "changeAssetScheme"."metadata" AS "changeAssetScheme.metadata", "changeAssetScheme"."approver" AS "changeAssetScheme.approver",
    "changeAssetScheme"."registrar" AS "changeAssetScheme.registrar", "changeAssetScheme"."allowedScriptHashes" AS "changeAssetScheme.allowedScriptHashes", "changeAssetScheme"."approvals" AS "changeAssetScheme.approvals",
    "changeAssetScheme"."seq" AS "changeAssetScheme.seq", "changeAssetScheme"."createdAt" AS "changeAssetScheme.createdAt", "changeAssetScheme"."updatedAt" AS "changeAssetScheme.updatedAt", "increaseAssetSupply"."transactionHash" AS "increaseAssetSupply.transactionHash",
    "increaseAssetSupply"."networkId" AS "increaseAssetSupply.networkId", "increaseAssetSupply"."shardId" AS "increaseAssetSupply.shardId", "increaseAssetSupply"."assetType" AS "increaseAssetSupply.assetType",
    "increaseAssetSupply"."approvals" AS "increaseAssetSupply.approvals", "increaseAssetSupply"."seq" AS "increaseAssetSupply.seq", "increaseAssetSupply"."lockScriptHash" AS "increaseAssetSupply.lockScriptHash",
    "increaseAssetSupply"."parameters" AS "increaseAssetSupply.parameters", "increaseAssetSupply"."supply" AS "increaseAssetSupply.supply", "increaseAssetSupply"."recipient" AS "increaseAssetSupply.recipient",
    "increaseAssetSupply"."createdAt" AS "increaseAssetSupply.createdAt", "increaseAssetSupply"."updatedAt" AS "increaseAssetSupply.updatedAt", "wrapCCC"."transactionHash" AS "wrapCCC.transactionHash", "wrapCCC"."shardId" AS "wrapCCC.shardId",
    "wrapCCC"."lockScriptHash" AS "wrapCCC.lockScriptHash", "wrapCCC"."parameters" AS "wrapCCC.parameters", "wrapCCC"."quantity" AS "wrapCCC.quantity", "wrapCCC"."recipient" AS "wrapCCC.recipient", "wrapCCC"."createdAt" AS "wrapCCC.createdAt",
    "wrapCCC"."updatedAt" AS "wrapCCC.updatedAt", "unwrapCCC"."transactionHash" AS "unwrapCCC.transactionHash", "unwrapCCC"."receiver" AS "unwrapCCC.receiver", "unwrapCCC"."burn" AS "unwrapCCC.burn", "unwrapCCC"."createdAt" AS "unwrapCCC.createdAt",
    "unwrapCCC"."updatedAt" AS "unwrapCCC.updatedAt", "pay"."transactionHash" AS "pay.transactionHash", "pay"."receiver" AS "pay.receiver", "pay"."quantity" AS "pay.quantity", "pay"."createdAt" AS "pay.createdAt", "pay"."updatedAt" AS "pay.updatedAt",
    "setRegularKey"."transactionHash" AS "setRegularKey.transactionHash", "setRegularKey"."key" AS "setRegularKey.key", "setRegularKey"."createdAt" AS "setRegularKey.createdAt", "setRegularKey"."updatedAt" AS "setRegularKey.updatedAt",
    "createShard"."transactionHash" AS "createShard.transactionHash", "createShard"."shardId" AS "createShard.shardId", "createShard"."users" AS "createShard.users", "createShard"."createdAt" AS "createShard.createdAt",
    "createShard"."updatedAt" AS "createShard.updatedAt", "setShardOwners"."transactionHash" AS "setShardOwners.transactionHash", "setShardOwners"."shardId" AS "setShardOwners.shardId", "setShardOwners"."owners" AS "setShardOwners.owners",
    "setShardOwners"."createdAt" AS "setShardOwners.createdAt", "setShardOwners"."updatedAt" AS "setShardOwners.updatedAt", "setShardUsers"."transactionHash" AS "setShardUsers.transactionHash", "setShardUsers"."shardId" AS "setShardUsers.shardId",
    "setShardUsers"."users" AS "setShardUsers.users", "setShardUsers"."createdAt" AS "setShardUsers.createdAt", "setShardUsers"."updatedAt" AS "setShardUsers.updatedAt", "store"."transactionHash" AS "store.transactionHash", "store"."content" AS "store.content",
    "store"."certifier" AS "store.certifier", "store"."signature" AS "store.signature", "store"."createdAt" AS "store.createdAt", "store"."updatedAt" AS "store.updatedAt", "remove"."transactionHash" AS "remove.transactionHash", "remove"."textHash" AS "remove.textHash",
    "remove"."signature" AS "remove.signature", "remove"."createdAt" AS "remove.createdAt", "remove"."updatedAt" AS "remove.updatedAt", "custom"."transactionHash" AS "custom.transactionHash", "custom"."handlerId" AS "custom.handlerId",
    "custom"."content" AS "custom.content", "custom"."createdAt" AS "custom.createdAt", "custom"."updatedAt" AS "custom.updatedAt" FROM "Transactions" AS "Transaction"
LEFT OUTER JOIN "MintAssets" AS "mintAsset" ON "Transaction"."hash" = "mintAsset"."transactionHash"
LEFT OUTER JOIN "TransferAssets" AS "transferAsset" ON "Transaction"."hash" = "transferAsset"."transactionHash"
LEFT OUTER JOIN "ComposeAssets" AS "composeAsset" ON "Transaction"."hash" = "composeAsset"."transactionHash"
LEFT OUTER JOIN "DecomposeAssets" AS "decomposeAsset" ON "Transaction"."hash" = "decomposeAsset"."transactionHash"
LEFT OUTER JOIN "ChangeAssetSchemes" AS "changeAssetScheme" ON "Transaction"."hash" = "changeAssetScheme"."transactionHash"
LEFT OUTER JOIN "IncreaseAssetSupplies" AS "increaseAssetSupply" ON "Transaction"."hash" = "increaseAssetSupply"."transactionHash"
LEFT OUTER JOIN "WrapCCCs" AS "wrapCCC" ON "Transaction"."hash" = "wrapCCC"."transactionHash"
LEFT OUTER JOIN "UnwrapCCCs" AS "unwrapCCC" ON "Transaction"."hash" = "unwrapCCC"."transactionHash"
LEFT OUTER JOIN "Pays" AS "pay" ON "Transaction"."hash" = "pay"."transactionHash"
LEFT OUTER JOIN "SetRegularKeys" AS "setRegularKey" ON "Transaction"."hash" = "setRegularKey"."transactionHash"
LEFT OUTER JOIN "CreateShards" AS "createShard" ON "Transaction"."hash" = "createShard"."transactionHash"
LEFT OUTER JOIN "SetShardOwners" AS "setShardOwners" ON "Transaction"."hash" = "setShardOwners"."transactionHash"
LEFT OUTER JOIN "SetShardUsers" AS "setShardUsers" ON "Transaction"."hash" = "setShardUsers"."transactionHash"
LEFT OUTER JOIN "Stores" AS "store" ON "Transaction"."hash" = "store"."transactionHash"
LEFT OUTER JOIN "Removes" AS "remove" ON "Transaction"."hash" = "remove"."transactionHash"
LEFT OUTER JOIN "Customs" AS "custom" ON "Transaction"."hash" = "custom"."transactionHash"
WHERE "Transaction"."hash" IN ('84ecfe11d09b7b1c5fa86151aca5ddd5a4f3527d382901e0ee488b654926b68d', '5e8a58c563ac53ec32b684df2ff82a16678257c71f7d0650b11e82402706026c', '512cb27f0fab399ed9c716eafa42382d82f9a40c4c90753bcf2e0c109d43fbb6', '381fa9a3b9473fa23f0b5c3d4e09d195ab697ab0a365bf54937c3ded7fda081c', 'd19f93f4fc7b704fef96eadfb7a47d1e3a4c71268d0cd0f325b9c2fd94c43b3d', '815bc6332cfdb436d4f7bf2417d2ca2223f8efecc80a65da8a0196b782fd2e87', '604e80e3df71fd53361dfb9820a351bff2276e1687e4dc2db278d28f89450a3e', 'f439bfcd8d8721e5bf051e046026bd5b0eebde964444588c66aebc07717cc9f2', 'aab5d29bffcfff6c02d80e7962dd5e99724cf29565ea969198ff2eab5995b1e4', 'f9bad3cde8571eb18fc8ef3d6c82a8520839cc27156d15d357e3d31ae05dc0dc', '1acda715273535d2d9f40cdcc7bd41ca13bb1efeed78ee2f6e9cee619ef0a360', '6114f446baf0634f7bb1d2e344e3c045392fbf840acfc3f6b7d249d28a9df09f', '89bd4e20da9c624f3ae09dc151ed2db16117b0e1c161cef921bd0cd350fc9a03', 'cc0e031fb71905e09056ee49407ad28beb5096391668c77f6544f262878acdff', '08ec172ae4795fa9a6ef957ed2513eecb2866fad76f2d4c0f882f26dff741a39') AND "Transaction"."isPending" = false ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC;
Limit  (cost=49681.35..49681.38 rows=15 width=73)
   ->  Sort  (cost=49681.35..51085.55 rows=561681 width=73)
         Sort Key: "blockNumber" DESC, "transactionIndex" DESC
         ->  Seq Scan on "Transactions" "Transaction"  (cost=0.00..35900.81 rows=561681 width=73)
               Filter: (NOT "isPending")
Sort  (cost=1182.22..1182.25 rows=15 width=17217)
   Sort Key: "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
   ->  Hash Left Join  (cost=929.07..1181.92 rows=15 width=17217)
         Hash Cond: (("Transaction".hash)::text = (remove."transactionHash")::text)
         ->  Merge Left Join  (cost=917.94..1170.76 rows=15 width=15653)
               Merge Cond: (("Transaction".hash)::text = (custom."transactionHash")::text)
               ->  Nested Loop Left Join  (cost=905.10..1157.52 rows=15 width=14601)
                     ->  Nested Loop Left Join  (cost=904.68..1030.92 rows=15 width=14471)
                           ->  Merge Left Join  (cost=904.27..904.50 rows=15 width=13421)
                                 Merge Cond: (("Transaction".hash)::text = (store."transactionHash")::text)
                                 ->  Sort  (cost=893.23..893.27 rows=15 width=11341)
                                       Sort Key: "Transaction".hash
                                       ->  Hash Left Join  (cost=811.60..892.94 rows=15 width=11341)
                                             Hash Cond: (("Transaction".hash)::text = ("setShardUsers"."transactionHash")::text)
                                             ->  Hash Left Join  (cost=798.67..879.97 rows=15 width=10773)
                                                   Hash Cond: (("Transaction".hash)::text = ("setShardOwners"."transactionHash")::text)
                                                   ->  Hash Left Join  (cost=785.75..867.01 rows=15 width=10205)
                                                         Hash Cond: (("Transaction".hash)::text = ("createShard"."transactionHash")::text)
                                                         ->  Hash Right Join  (cost=772.82..854.04 rows=15 width=9637)
                                                               Hash Cond: (("setRegularKey"."transactionHash")::text = ("Transaction".hash)::text)
                                                               ->  Seq Scan on "SetRegularKeys" "setRegularKey"  (cost=0.00..74.88 rows=1688 width=213)
                                                               ->  Hash  (cost=772.64..772.64 rows=15 width=9424)
                                                                     ->  Nested Loop Left Join  (cost=13.70..772.64 rows=15 width=9424)
                                                                           ->  Nested Loop Left Join  (cost=13.29..646.22 rows=15 width=9198)
                                                                                 ->  Nested Loop Left Join  (cost=13.01..525.76 rows=15 width=8850)
                                                                                       ->  Hash Left Join  (cost=12.73..405.25 rows=15 width=8056)
                                                                                             Hash Cond: (("Transaction".hash)::text = ("decomposeAsset"."transactionHash")::text)
                                                                                             ->  Nested Loop Left Join  (cost=1.38..393.86 rows=15 width=6912)
                                                                                                   Join Filter: (("Transaction".hash)::text = ("composeAsset"."transactionHash")::text)
                                                                                                   ->  Nested Loop Left Join  (cost=1.38..381.49 rows=15 width=2584)
                                                                                                         ->  Nested Loop Left Join  (cost=0.96..254.93 rows=15 width=1009)
                                                                                                               ->  Index Scan using "Transactions_pkey" on "Transactions" "Transaction"  (cost=0.5..128.51 rows=15 width=458)
                                                                                                                     Index Cond: ((hash)::text = ANY ('{...}'::text[]))
                                                                                                                     Filter: (NOT "isPending")
                                                                                                               ->  Index Scan using "MintAssets_pkey" on "MintAssets" "mintAsset"  (cost=0.41..8.43
 rows=1 width=551)
                                                                                                                     Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
                                                                                                         ->  Index Scan using "TransferAssets_pkey" on "TransferAssets" "transferAsset"  (cost=0.42
..8.44 rows=1 width=1575)
                                                                                                               Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
                                                                                                   ->  Materialize  (cost=0.00..10.15 rows=10 width=4328)
                                                                                                         ->  Seq Scan on "ComposeAssets" "composeAsset"  (cost=0.00..10.10 rows=10 width=4328)
                                                                                             ->  Hash  (cost=10.60..10.60 rows=60 width=1144)
                                                                                                   ->  Seq Scan on "DecomposeAssets" "decomposeAsset"  (cost=0.00..10.60 rows=60 width=1144)
                                                                                       ->  Index Scan using "ChangeAssetSchemes_pkey" on "ChangeAssetSchemes" "changeAssetScheme"  (cost=0.28..8.03
 rows=1 width=794)
                                                                                             Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
                                                                                 ->  Index Scan using "IncreaseAssetSupplies_pkey" on "IncreaseAssetSupplies" "increaseAssetSupply"  (cost=0.28..8.
03 rows=1 width=348)
                                                                                       Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
                                                                           ->  Index Scan using "WrapCCCs_pkey" on "WrapCCCs" "wrapCCC"  (cost=0.41..8.43 rows=1 width=226)
                                                                                 Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
                                                         ->  Hash  (cost=11.30..11.30 rows=130 width=568)
                                                               ->  Seq Scan on "CreateShards" "createShard"  (cost=0.00..11.30 rows=130 width=568)
                                                   ->  Hash  (cost=11.30..11.30 rows=130 width=568)
                                                         ->  Seq Scan on "SetShardOwners" "setShardOwners"  (cost=0.00..11.30 rows=130 width=568)
                                             ->  Hash  (cost=11.30..11.30 rows=130 width=568)
                                                   ->  Seq Scan on "SetShardUsers" "setShardUsers"  (cost=0.00..11.30 rows=130 width=568)
                                 ->  Sort  (cost=11.04..11.11 rows=30 width=2080)
                                       Sort Key: store."transactionHash"
                                       ->  Seq Scan on "Stores" store  (cost=0.00..10.30 rows=30 width=2080)
                           ->  Index Scan using "UnwrapCCCs_pkey" on "UnwrapCCCs" "unwrapCCC"  (cost=0.41..8.43 rows=1 width=1050)
                                 Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
                     ->  Index Scan using "Pays_pkey" on "Pays" pay  (cost=0.42..8.44 rows=1 width=130)
                           Index Cond: (("Transaction".hash)::text = ("transactionHash")::text)
               ->  Sort  (cost=12.85..13.02 rows=70 width=1052)
                     Sort Key: custom."transactionHash"
                     ->  Seq Scan on "Customs" custom  (cost=0.00..10.70 rows=70 width=1052)
         ->  Hash  (cost=10.50..10.50 rows=50 width=1564)
               ->  Seq Scan on "Removes" remove  (cost=0.00..10.50 rows=50 width=1564)

Still sorting is the most expensive operator, but it's much cheaper than the previous one because the width is much smaller(17217 vs. 73).

@sgkim126
Copy link
Contributor Author

sgkim126 commented May 8, 2019

However, curl -X GET "http://localhost:9001/api/tx?address={address}" -H "accept: application/json" is still slow.
I'm fixing it.

@sgkim126
Copy link
Contributor Author

sgkim126 commented May 8, 2019 •
edited
Loading

This patch make curl -X GET "http://localhost:9001/api/tx?address={address}" -H "accept: application/json" generate the below query to get the transaction hash.

SELECT "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."blockHash", "Transaction"."tracker", "Transaction"."transactionIndex", "Transaction"."type", "Transaction"."seq", "Transaction"."fee", "Transaction"."networkId",
    "Transaction"."sig", "Transaction"."signer", "Transaction"."errorHint", "Transaction"."timestamp", "Transaction"."isPending", "Transaction"."pendingTimestamp", "Transaction"."createdAt", "Transaction"."updatedAt"
FROM "Transactions" AS "Transaction"
WHERE "Transaction"."isPending" = false
    AND (
        SELECT "transactionHash"
        FROM "AddressLogs" AS "addressLogs"
        WHERE ("addressLogs"."address" = 'wccq92pg86am6jrwl2gdlckdakwdm70ff5gs5w8ryc0' AND "addressLogs"."transactionHash" = "Transaction"."hash") LIMIT 1
    ) IS NOT NULL
ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC

but it can be optimized as the below.

SELECT "Transaction".*
FROM (
    SELECT "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."transactionIndex"
    FROM "Transactions" AS "Transaction"
    INNER JOIN "AddressLogs" AS "addressLogs"
    ON "Transaction"."hash" = "addressLogs"."transactionHash"
    WHERE "Transaction"."isPending" = false
        AND "addressLogs"."address" = 'wccq959puf2uj2587x86am0ycxeqjqgnrm9uy7qyjhr'
) AS "Transaction"
ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
LIMIT 15 OFFSET 0

I'm finding a way to execute this query in sequelize.
=== edited ===
The above query is wrong because addressLogs can have multiple rows that has the same transactionHash.
We should find a way to execute the below query

SELECT "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."transactionIndex"
FROM "Transactions" AS "Transaction"
INNER JOIN "AddressLogs" AS "addressLogs" ON "Transaction"."hash" = "addressLogs"."transactionHash" AND "addressLogs"."address" = 'wccq92pg86am6jrwl2gdlckdakwdm70ff5gs5w8ryc0'
WHERE "Transaction"."isPending" = false
GROUP BY "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."transactionIndex"
ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
LIMIT 15 OFFSET 0;

@sgkim126
Copy link
Contributor Author

sgkim126 commented May 8, 2019 •
edited
Loading

Now, curl -X GET "http://localhost:9001/api/tx?address={address}" -H "accept: application/json" generates the below query to get transaction hashes.

SELECT "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."transactionIndex"
FROM "Transactions" AS "Transaction"
INNER JOIN "AddressLogs" AS "addressLogs" ON "Transaction"."hash" = "addressLogs"."transactionHash" AND "addressLogs"."address" = 'wccq92pg86am6jrwl2gdlckdakwdm70ff5gs5w8ryc0'
WHERE "Transaction"."isPending" = false
GROUP BY "Transaction"."hash", "Transaction"."blockNumber", "Transaction"."transactionIndex"
ORDER BY "Transaction"."blockNumber" DESC, "Transaction"."transactionIndex" DESC
LIMIT 15 OFFSET 0;

@sgkim126 sgkim126 changed the title [WIP] Optimize the API to get transactions Optimize the API to get transactions May 8, 2019
@sgkim126
Copy link
Contributor Author

sgkim126 commented May 8, 2019

@joojis Please review it.

Optimize the API to get transactions
1893b00 
The previous query joins all rows of tables to calculate offset even
though it needs only itemsPerPage rows.

This patch splits the query to get transactions. The first query gets
the transaction hash that the second query selects. It helps the second
query not to scan all the table of the rows of included tables.
joojis
joojis approved these changes May 9, 2019
View reviewed changes
Copy link
Contributor

@joojis joojis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@joojis joojis merged commit a5607cc into CodeChain-io:master May 10, 2019
@sgkim126 sgkim126 mentioned this pull request May 10, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joojis joojis joojis approved these changes

Assignees

No one assigned

Labels

optimization This makes the system faster

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sgkim126 @joojis