New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPU spikes: Lack of Indexing on OWTF db? #444

Closed
viyatb opened this Issue May 1, 2015 · 5 comments

Comments

Projects
None yet
4 participants
@viyatb
Member

viyatb commented May 1, 2015

@7a :

OWTF seems to struggle in huge assessments, below is after running Dirbuster, with just 2 workers, with almost NO urls found, this CPU spikes suggest that OWTF would probably benefit from indexing the DB.

The scope in question was a FULL web scan (i.e. ALL web plugins) of 35 URLs, towards the end.

@7a

This comment has been minimized.

Show comment
Hide comment
@7a

7a Jul 1, 2015

Member

Confirmed as lack of indexing: adding the following index eliminated ALL CPU spikes "idle in transaction", "SELECT", etc.

TODO: The db schema creation scripts, perhaps even the installer (which will be re-run after upgrading, maybe), should fix the DB schema, maybe OWTF should have backwards checks like this on start

n
Name Definition Constraints Clustered? Actions Comment
transaction_id_idx CREATE INDEX transaction_id_idx ON transaction_grep_association USING btree (transaction_id)

Member

7a commented Jul 1, 2015

Confirmed as lack of indexing: adding the following index eliminated ALL CPU spikes "idle in transaction", "SELECT", etc.

TODO: The db schema creation scripts, perhaps even the installer (which will be re-run after upgrading, maybe), should fix the DB schema, maybe OWTF should have backwards checks like this on start

n
Name Definition Constraints Clustered? Actions Comment
transaction_id_idx CREATE INDEX transaction_id_idx ON transaction_grep_association USING btree (transaction_id)

@DePierre

This comment has been minimized.

Show comment
Hide comment
@DePierre

DePierre Jul 26, 2015

Contributor

@tunnelshade Should we include such fix? What are the side effects?

Contributor

DePierre commented Jul 26, 2015

@tunnelshade Should we include such fix? What are the side effects?

@tunnelshade

This comment has been minimized.

Show comment
Hide comment
@tunnelshade

tunnelshade Jul 26, 2015

Member

Yes, indexing will solve this for now as Abe suggested. Add indexing to the column of the transaction_grep_association table

Member

tunnelshade commented Jul 26, 2015

Yes, indexing will solve this for now as Abe suggested. Add indexing to the column of the transaction_grep_association table

@7a

This comment has been minimized.

Show comment
Hide comment
@7a

7a Aug 31, 2015

Member

Can somebody please add this to the DB init scripts? the index is still missing on develop and the spikes are therefore happening.

Simply create a btree index on the transaction_id column of the transaction_grep_association table, as above, and the spike is gone, as I troubleshooted and fixed 2 months ago above! :)

NOTE: This takes forever using the postgress DB UI, if using phppgadmin, you need to do this on an empty database to have a chance to see the full command generated.

Member

7a commented Aug 31, 2015

Can somebody please add this to the DB init scripts? the index is still missing on develop and the spikes are therefore happening.

Simply create a btree index on the transaction_id column of the transaction_grep_association table, as above, and the spike is gone, as I troubleshooted and fixed 2 months ago above! :)

NOTE: This takes forever using the postgress DB UI, if using phppgadmin, you need to do this on an empty database to have a chance to see the full command generated.

tunnelshade added a commit that referenced this issue Sep 3, 2015

viyatb added a commit that referenced this issue Oct 21, 2015

@viyatb

This comment has been minimized.

Show comment
Hide comment
@viyatb

viyatb Dec 6, 2015

Member

Confirmed from @7a : the last referenced commit fixes the problem :)

Member

viyatb commented Dec 6, 2015

Confirmed from @7a : the last referenced commit fixes the problem :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment