Log profiling data for visualization #412

domoritz · 2014-02-19T04:37:50Z

Tasks:

store all the performance log in the CSV format
If the query is in profiling mode, insert the performance log into master as table public#log#query-ID
Deprecate the python extract script, let the REST api return visualization data from master database

Unresolved issues:

Sqlite write lock will prevent the new logging actions.

… logs into myria

domoritz · 2014-02-19T04:39:55Z

@stechu This branch uses CSV so we should add the new log storing and copying mechanism in here.

stechu · 2014-02-19T09:24:16Z

@domoritz We do not have a master database yet, which means there is no way to do the insertion on master. So I will insert the logs into worker database first.

domoritz · 2014-02-19T18:12:52Z

@stechu Does that mean we need the distributed insert?

When someone needs the data, they will have to submit a query to get the logs to the master right? We could write the logs to a sqlite database on the master for now so that we don't have to pull every time.

…coding

…uples.

…arameters that are passed down an expression into one class.

Conflicts: src/edu/washington/escience/myria/expression/AndExpression.java src/edu/washington/escience/myria/expression/BinaryExpression.java src/edu/washington/escience/myria/expression/ExpressionOperator.java src/edu/washington/escience/myria/expression/NotExpression.java src/edu/washington/escience/myria/expression/OrExpression.java test/edu/washington/escience/myria/operator/apply/ApplyTest.java

domoritz · 2014-03-20T19:06:24Z

Thanks. Works like a charm. Removed WIP.

dhalperi · 2014-03-20T21:13:30Z

src/edu/washington/escience/myria/parallel/ProfilingLogger.java

+    try {
+      accessMethod.createIndexes(MyriaConstants.PROFILING_RELATION, MyriaConstants.PROFILING_SCHEMA, index.build());
+    } catch (DbException e) {
+      LOGGER.info(String.format("Tried to create index for profiling logs: %s", e.getMessage()));


Would have expected something like

LOGGER.error("Couldn't create index for profiling logs:", e);

Seems worth higher priority than info

Uses [the below][1] to automatically format the error

[1] : http://www.slf4j.org/api/org/slf4j/Logger.html#error(java.lang.String, java.lang.Throwable)

I'm totally not happy with this anyway. I would like to be able to use CREATE INDEX IF NOT EXISTS but there is no such command in Postgres and MySQL.

I changed the code to log an error instead of just info.

Gotcha. And we don't want to create the table when the worker is run the first time, probably because we don't have any mechanism for that?

We could see whether the table exists and create only create the table + index in this case. In any case, I am not happy with this but I don't think it's worth trying to make this perfect.

dhalperi · 2014-03-20T22:47:33Z

src/edu/washington/escience/myria/Tuple.java

+   * @param statement the JDBC statement
+   * @throws SQLException when setting values in the statement fails
+   */
+  public void getIntoJdbc(final PreparedStatement statement) throws SQLException {


Is this function used? I noticed it doesn't @Override anything but also getIntoJdbc does not appear elsewhere in the diff.

…tuples to store profiling data.

dhalperi · 2014-03-20T23:04:25Z

src/edu/washington/escience/myria/api/encoding/QueryStatusEncoding.java

@@ -26,11 +26,13 @@ public QueryStatusEncoding(final long queryId) {
   * @return a QueryStatusEncoding object containing the submitted query, with the submit time set to
   *         DateTimeUtils.nowInISO8601().
   */
-  public static QueryStatusEncoding submitted(final String rawQuery, final String logicalRa, final Object physicalPlan) {
+  public static QueryStatusEncoding submitted(final String rawQuery, final String logicalRa, final Object physicalPlan,
+      final Boolean profilingMode) {


missing @param?

Where? Are you looking at a commit or the whole patch?

I put it in yesterday

4304c9e

dhalperi · 2014-03-20T23:53:49Z

I merged master because that one stupid test keeps failing, so I merged in the test-fix. :)

coveralls · 2014-03-21T00:18:36Z

Coverage decreased (-0.21%) when pulling 37844bc on domoritz-visualization into 38670e9 on master.

Use Preconditions.checkArgument and disambiguate error messages.

This reverts commit b9dc910. The logic was wrong -- the 0 is misinterpreted. Instead, introduce the static int[][]{{0}} that is the collector partition->channel mapping.

They don't pass reliably enough, so I'm disabling them until we have someone who's interested in fixing them.

dhalperi · 2014-03-21T20:04:03Z

Looks like there must still be some race conditions or other timeout issues in the server, because the json submit test fails but now with timeout exceptions. Hmm

coveralls · 2014-03-21T20:52:19Z

Coverage decreased (-0.23%) when pulling f50f305 on domoritz-visualization into 38670e9 on master.

Log profiling data for visualization

domoritz added 7 commits February 16, 2014 17:18

Fix remove_deployment.py

8bcf311

Merge branch 'CSV_standard_support' into domoritz-visualization

4ef53b9

Update json query plans

a1b9641

Fix get_logs file

bf609b8

Upgrade slf4j

cf85d4a

Use CSV for performance logging so that we can use filescan to ingest…

c813e19

… logs into myria

Merge remote-tracking branch 'origin' into domoritz-visualization

259f287

domoritz added the WIP label Feb 19, 2014

stechu and others added 19 commits February 19, 2014 14:22

add myria constants for profiling log.

b6bbd31

add REST api for collecting logs, add profiling mode to QueryStatusEn…

f7917e1

…coding

fix javadoc.

898317a

fix test.

3c4b4c5

Write profiling data directly into the (postgres) database

5f1590a

Fix merge

8741729

More meaningful data for profiling log relation

6eb6182

add profiling mode to master catalog

5ab87db

Add stored relation

ef89fae

Use importDataset to create the profiling data relation.

3ddddd2

let server know which dbms is used

bd55fd7

Make the profiling logger a singleton. Buffer performance measurements.

b2532ba

Directly insert into database and don't go through tuple batches or t…

dae0a1a

…uples.

Throw exception when somebody tries the profiling mode with sqlite.

6e1ed9c

Add expression operator to get the worker id. I also refactored the p…

ceb9193

…arameters that are passed down an expression into one class.

Remove extract profiling data python script

6836d9e

Get nano time in logger

609c40e

Add worker id to profiling logs

0c3434e

Store worker id and not channel in sent logs

1be250d

domoritz changed the title ~~[WIP] Log profiling data for visualization~~ Log profiling data for visualization Mar 20, 2014

domoritz added 2 commits March 20, 2014 12:15

make getOutputIDs protected.

784b761

Refactor for loop

eae23ea

dhalperi reviewed Mar 20, 2014
View reviewed changes

Log error when index cannot be created.

7161cbb

dhalperi reviewed Mar 20, 2014
View reviewed changes

Remove unused function. It is not used any more because we don't use …

ee82f02

…tuples to store profiling data.

dhalperi reviewed Mar 20, 2014
View reviewed changes

Merge branch 'master' into domoritz-visualization

37844bc

dhalperi and others added 9 commits March 20, 2014 17:43

Server: cleanup argument checking

2c1b94d

Use Preconditions.checkArgument and disambiguate error messages.

disable flow control test in travis

8ac16a7

typo fix

a6b649d

Revert "Use myria constant for master worker id"

fe97f2c

This reverts commit b9dc910. The logic was wrong -- the 0 is misinterpreted. Instead, introduce the static int[][]{{0}} that is the collector partition->channel mapping.

QueryStatusEncoding: add missing param

4304c9e

FailureTest: disable on travis

ea0ddb3

They don't pass reliably enough, so I'm disabling them until we have someone who's interested in fixing them.

Typo and missing final

0ded68a

Remove unnecessary if

58c3f42

Default profiling mode to false in catalog db

f50f305

dhalperi added a commit that referenced this pull request Mar 23, 2014

Merge pull request #412 from uwescience/domoritz-visualization

8064e2e

Log profiling data for visualization

dhalperi merged commit 8064e2e into master Mar 23, 2014

dhalperi deleted the domoritz-visualization branch March 23, 2014 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log profiling data for visualization #412

Log profiling data for visualization #412

domoritz commented Feb 19, 2014

domoritz commented Feb 19, 2014

stechu commented Feb 19, 2014

domoritz commented Feb 19, 2014

domoritz commented Mar 20, 2014

dhalperi Mar 20, 2014

domoritz Mar 20, 2014

dhalperi Mar 20, 2014

domoritz Mar 20, 2014

dhalperi Mar 20, 2014

dhalperi Mar 20, 2014

domoritz Mar 21, 2014

dhalperi Mar 21, 2014

domoritz Mar 21, 2014

dhalperi commented Mar 20, 2014

coveralls commented Mar 21, 2014

dhalperi commented Mar 21, 2014

coveralls commented Mar 21, 2014

Log profiling data for visualization #412

Log profiling data for visualization #412

Conversation

domoritz commented Feb 19, 2014

domoritz commented Feb 19, 2014

stechu commented Feb 19, 2014

domoritz commented Feb 19, 2014

domoritz commented Mar 20, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented Mar 20, 2014

coveralls commented Mar 21, 2014

dhalperi commented Mar 21, 2014

coveralls commented Mar 21, 2014