Skip to content

perf: database optimization #3605

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
faustbrian merged 7 commits into from
Mar 19, 2020
Merged

perf: database optimization #3605

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6cecacc
refactor(core-api): remove not needed sorting and filtering
air1one Mar 17, 2020
94c06fe
perf: add necessary indexes for db performance
air1one Mar 17, 2020
da183aa
refactor: migrations sql file
air1one Mar 17, 2020
9c25244
test: fix failing tests
air1one Mar 17, 2020
2432c9a
chore: default api rate limit to 100/min
air1one Mar 18, 2020
44ed22c
test(functional): set up CORE_API_RATE_LIMIT_USER_LIMIT
air1one Mar 18, 2020
5239b9f
test(functional): api rate limit to 300 for tests
air1one Mar 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .github/workflows/functional.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

bridgechain-resignation:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -104,6 +105,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

bridgechain-update:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -152,6 +154,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

business-registration:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -200,6 +203,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

business-resignation:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -248,6 +252,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

business-update:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -296,6 +301,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

delegate-registration:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -344,6 +350,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

delegate-resignation:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -392,6 +399,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

htlc-claim:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -440,6 +448,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

htlc-lock:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -488,6 +497,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

htlc-refund:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -536,6 +546,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

ipfs:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -584,6 +595,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

multi-payment:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -632,6 +644,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

multi-signature-registration:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -680,6 +693,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

second-signature-registration:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -728,6 +742,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

transfer:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -776,6 +791,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

vote:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -824,6 +840,7 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300

pool:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -872,3 +889,4 @@ jobs:
POSTGRES_USER: ark
POSTGRES_PASSWORD: password
POSTGRES_DB: ark_unitnet
CORE_API_RATE_LIMIT_USER_LIMIT: 300
71 changes: 0 additions & 71 deletions __tests__/integration/core-api/handlers/blocks.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,23 +247,6 @@ describe("API 2.0 - Blocks", () => {
await databaseService.deleteBlocks([block2.data]); // reset to genesis block
});

it("should POST a search for blocks with the exact specified payloadHash", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
payloadHash: genesisBlock.payloadHash,
});
expect(response).toBeSuccessfulResponse();
expect(response.data.data).toBeArray();

expect(response.data.data).toHaveLength(1);

const block = response.data.data[0];
utils.expectBlock(block);
expect(block.id).toBe(genesisBlock.id);
expect(block.payload.length).toBe(genesisBlock.payloadLength);
expect(block.payload.hash).toBe(genesisBlock.payloadHash);
});

it("should POST a search for blocks with the exact specified generatorPublicKey", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
Expand All @@ -280,22 +263,6 @@ describe("API 2.0 - Blocks", () => {
expect(block.generator.publicKey).toBe(genesisBlock.generatorPublicKey);
});

it("should POST a search for blocks with the exact specified blockSignature", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
blockSignature: genesisBlock.blockSignature,
});
expect(response).toBeSuccessfulResponse();
expect(response.data.data).toBeArray();

expect(response.data.data).toHaveLength(1);

const block = response.data.data[0];
utils.expectBlock(block);
expect(block.id).toBe(genesisBlock.id);
expect(block.signature).toBe(genesisBlock.blockSignature);
});

it("should POST a search for blocks with the exact specified timestamp", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
Expand Down Expand Up @@ -491,44 +458,6 @@ describe("API 2.0 - Blocks", () => {
expect(+block.forged.reward).toBe(+genesisBlock.reward.toFixed());
});

it("should POST a search for blocks with the exact specified payloadLength", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
payloadLength: {
from: genesisBlock.payloadLength,
to: genesisBlock.payloadLength,
},
});
expect(response).toBeSuccessfulResponse();
expect(response.data.data).toBeArray();

expect(response.data.data).toHaveLength(1);

const block = response.data.data[0];
utils.expectBlock(block);
expect(block.id).toBe(genesisBlock.id);
expect(block.payload.length).toBe(genesisBlock.payloadLength);
});

it("should POST a search for blocks with the specified payloadLength range", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
payloadLength: {
from: genesisBlock.payloadLength,
to: genesisBlock.payloadLength,
},
});
expect(response).toBeSuccessfulResponse();
expect(response.data.data).toBeArray();

expect(response.data.data).toHaveLength(1);

const block = response.data.data[0];
utils.expectBlock(block);
expect(block.id).toBe(genesisBlock.id);
expect(block.payload.length).toBe(genesisBlock.payloadLength);
});

it("should POST a search for blocks with the wrong specified version", async () => {
const response = await utils.request("POST", "blocks/search", {
id: genesisBlock.id,
Expand Down
2 changes: 1 addition & 1 deletion packages/core-api/src/defaults.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export const defaults = {
rateLimit: {
enabled: !process.env.CORE_API_RATE_LIMIT,
pathLimit: false,
userLimit: process.env.CORE_API_RATE_LIMIT_USER_LIMIT || 300,
userLimit: process.env.CORE_API_RATE_LIMIT_USER_LIMIT || 100,
userCache: {
expiresIn: process.env.CORE_API_RATE_LIMIT_USER_EXPIRES || 60000,
},
Expand Down
15 changes: 0 additions & 15 deletions packages/core-api/src/handlers/blocks/schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,7 @@ export const index: object = {
reward: Joi.number()
.integer()
.min(0),
payloadLength: Joi.number()
.integer()
.positive(),
payloadHash: Joi.string().hex(),
generatorPublicKey: publicKey,
blockSignature: Joi.string().hex(),
transform: Joi.bool().default(true),
},
},
Expand Down Expand Up @@ -112,9 +107,7 @@ export const search: object = {
.integer()
.min(0),
previousBlock: blockId,
payloadHash: Joi.string().hex(),
generatorPublicKey: publicKey,
blockSignature: Joi.string().hex(),
timestamp: Joi.object().keys({
from: Joi.number()
.integer()
Expand Down Expand Up @@ -163,13 +156,5 @@ export const search: object = {
.integer()
.min(0),
}),
payloadLength: Joi.object().keys({
from: Joi.number()
.integer()
.min(0),
to: Joi.number()
.integer()
.min(0),
}),
},
};
8 changes: 0 additions & 8 deletions packages/core-api/src/handlers/shared/iteratees.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
export const blockIteratees = [
"height",
"id",
"numberOfTransactions",
"payloadHash",
"payloadLength",
"previousBlock",
"reward",
"timestamp",
"totalAmount",
Expand Down Expand Up @@ -54,12 +50,8 @@ export const peerIteratees = ["height", "ip", "latency", "version"];

export const transactionIteratees = [
"amount",
"blockId",
"fee",
"id",
"nonce",
"recipientId",
"senderPublicKey",
"timestamp",
"type",
"typeGroup",
Expand Down
30 changes: 30 additions & 0 deletions ...e-database-postgres/src/migrations/20200317000000-add-blocks-and-transactions-indexes.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
CREATE INDEX IF NOT EXISTS "transactions_asset" ON ${schema~}.transactions USING gin (asset);
CREATE INDEX IF NOT EXISTS "transactions_amount" ON ${schema~}.transactions ("amount");
CREATE INDEX IF NOT EXISTS "transactions_fee" ON ${schema~}.transactions ("fee");
CREATE INDEX IF NOT EXISTS "transactions_nonce_idx" ON ${schema~}.transactions ("nonce");
CREATE INDEX IF NOT EXISTS "transactions_vendor_field" ON ${schema~}.transactions ("vendor_field");
CREATE INDEX IF NOT EXISTS "transactions_version" ON ${schema~}.transactions ("version");

CREATE INDEX IF NOT EXISTS "transactions_amount_sequence" ON ${schema~}.transactions ("amount","sequence");
CREATE INDEX IF NOT EXISTS "transactions_fee_sequence" ON ${schema~}.transactions ("fee","sequence");
CREATE INDEX IF NOT EXISTS "transactions_nonce_sequence" ON ${schema~}.transactions ("nonce","sequence");
CREATE INDEX IF NOT EXISTS "transactions_timestamp_sequence" ON ${schema~}.transactions ("timestamp","sequence");
CREATE INDEX IF NOT EXISTS "transactions_type_sequence" ON ${schema~}.transactions ("type","sequence");
CREATE INDEX IF NOT EXISTS "transactions_type_group_sequence" ON ${schema~}.transactions ("type_group","sequence");
CREATE INDEX IF NOT EXISTS "transactions_vendor_field_sequence" ON ${schema~}.transactions ("vendor_field","sequence");
CREATE INDEX IF NOT EXISTS "transactions_version_sequence" ON ${schema~}.transactions ("version","sequence");

CREATE INDEX IF NOT EXISTS "transactions_amount_asc_sequence" ON ${schema~}.transactions ("amount" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_fee_asc_sequence" ON ${schema~}.transactions ("fee" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_nonce_asc_sequence" ON ${schema~}.transactions ("nonce" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_timestamp_asc_sequence" ON ${schema~}.transactions ("timestamp" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_type_asc_sequence" ON ${schema~}.transactions ("type" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_type_group_asc_sequence" ON ${schema~}.transactions ("type_group" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_vendor_field_asc_sequence" ON ${schema~}.transactions ("vendor_field" ASC,"sequence" DESC);
CREATE INDEX IF NOT EXISTS "transactions_version_asc_sequence" ON ${schema~}.transactions ("version" ASC,"sequence" DESC);

CREATE INDEX IF NOT EXISTS "blocks_number_of_transactions" ON ${schema~}.blocks ("number_of_transactions");
CREATE INDEX IF NOT EXISTS "blocks_reward" ON ${schema~}.blocks ("reward");
CREATE INDEX IF NOT EXISTS "blocks_total_amount" ON ${schema~}.blocks ("total_amount");
CREATE INDEX IF NOT EXISTS "blocks_total_fee" ON ${schema~}.blocks ("total_fee");
CREATE INDEX IF NOT EXISTS "blocks_version" ON ${schema~}.blocks ("version");