New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query not found (aka csv writing out of bound exception) #444

Closed
domoritz opened this Issue Mar 10, 2014 · 7 comments

Comments

Projects
None yet
2 participants
@domoritz
Member

domoritz commented Mar 10, 2014

After running the query successfully for around 100 times, I see the following error in the logs and from now on the query never finishes successfully.

ERROR 2014-03-09 21:49:20,218 [Pipeline executor#2] WorkerShortMessageProcessor - In receiving message query_id: 304
type: QUERY_START
, unknown query id: 304, current active queries are: [], query contained? null
ERROR 2014-03-09 21:49:20,317 [Pipeline executor#2] WorkerShortMessageProcessor - In receiving message query_id: 305
type: QUERY_START
, unknown query id: 305, current active queries are: [], query contained? null
ERROR 2014-03-09 21:49:20,524 [Pipeline executor#2] WorkerShortMessageProcessor - In receiving message query_id: 306
type: QUERY_START
, unknown query id: 306, current active queries are: [], query contained? null
ERROR 2014-03-09 21:49:20,674 [Pipeline executor#2] WorkerShortMessageProcessor - In receiving message query_id: 307
type: QUERY_START
, unknown query id: 307, current active queries are: [], query contained? null
ERROR 2014-03-09 21:49:21,199 [Pipeline executor#0] WorkerShortMessageProcessor - In receiving message query_id: 310
type: QUERY_START
, unknown query id: 310, current active queries are: [], query contained? null

The problem is that the problem disappears when I restart the server.

I know it's hard to reproduce but my hope is that someone has a tip for what to look into.

@domoritz domoritz added the Bug label Mar 10, 2014

@domoritz

This comment has been minimized.

Show comment
Hide comment
@domoritz

domoritz Mar 10, 2014

Member

Related to #412. Looks like its a problem with the csv headers.

ERROR 2014-03-10 13:16:38,422 [Master message processor#0] QuerySubTreeTask - Task failed to open because of exception:
java.lang.StringIndexOutOfBoundsException
        at java.lang.AbstractStringBuilder.delete(AbstractStringBuilder.java:698)
        at java.lang.StringBuilder.delete(StringBuilder.java:250)
        at org.supercsv.encoder.DefaultCsvEncoder.encode(DefaultCsvEncoder.java:42)
        at org.supercsv.io.AbstractCsvWriter.escapeString(AbstractCsvWriter.java:102)
        at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:196)
        at org.supercsv.io.CsvListWriter.write(CsvListWriter.java:87)
        at edu.washington.escience.myria.CsvTupleWriter.writeColumnHeaders(CsvTupleWriter.java:63)
        at edu.washington.escience.myria.operator.DataOutput.init(DataOutput.java:62)
        at edu.washington.escience.myria.operator.Operator.open(Operator.java:391)
        at edu.washington.escience.myria.parallel.QuerySubTreeTask.init(QuerySubTreeTask.java:629)
        at edu.washington.escience.myria.parallel.MasterQueryPartition.init(MasterQueryPartition.java:543)
        at edu.washington.escience.myria.parallel.Server$3.operationComplete(Server.java:1093)
        at edu.washington.escience.myria.parallel.QueryFutureListener.operationComplete(QueryFutureListener.java:43)
Member

domoritz commented Mar 10, 2014

Related to #412. Looks like its a problem with the csv headers.

ERROR 2014-03-10 13:16:38,422 [Master message processor#0] QuerySubTreeTask - Task failed to open because of exception:
java.lang.StringIndexOutOfBoundsException
        at java.lang.AbstractStringBuilder.delete(AbstractStringBuilder.java:698)
        at java.lang.StringBuilder.delete(StringBuilder.java:250)
        at org.supercsv.encoder.DefaultCsvEncoder.encode(DefaultCsvEncoder.java:42)
        at org.supercsv.io.AbstractCsvWriter.escapeString(AbstractCsvWriter.java:102)
        at org.supercsv.io.AbstractCsvWriter.writeRow(AbstractCsvWriter.java:196)
        at org.supercsv.io.CsvListWriter.write(CsvListWriter.java:87)
        at edu.washington.escience.myria.CsvTupleWriter.writeColumnHeaders(CsvTupleWriter.java:63)
        at edu.washington.escience.myria.operator.DataOutput.init(DataOutput.java:62)
        at edu.washington.escience.myria.operator.Operator.open(Operator.java:391)
        at edu.washington.escience.myria.parallel.QuerySubTreeTask.init(QuerySubTreeTask.java:629)
        at edu.washington.escience.myria.parallel.MasterQueryPartition.init(MasterQueryPartition.java:543)
        at edu.washington.escience.myria.parallel.Server$3.operationComplete(Server.java:1093)
        at edu.washington.escience.myria.parallel.QueryFutureListener.operationComplete(QueryFutureListener.java:43)
@domoritz

This comment has been minimized.

Show comment
Hide comment
@domoritz

domoritz Mar 10, 2014

Member

Looks like we have an issue with multithreading in supercsv http://sourceforge.net/p/supercsv/bugs/43/.

Member

domoritz commented Mar 10, 2014

Looks like we have an issue with multithreading in supercsv http://sourceforge.net/p/supercsv/bugs/43/.

@domoritz

This comment has been minimized.

Show comment
Hide comment
@domoritz

domoritz Mar 10, 2014

Member

Probably fixed via 178fa9b.

Member

domoritz commented Mar 10, 2014

Probably fixed via 178fa9b.

@dhalperi

This comment has been minimized.

Show comment
Hide comment
@dhalperi

dhalperi Mar 10, 2014

Member

Good, thanks. Can we spin that patch off as a bugfix PR, since presumably it affects all code not just vis?

In future, would appreciate a ref to this issue or to the supercsv bug thread in the bugfix commit as well as here.

Member

dhalperi commented Mar 10, 2014

Good, thanks. Can we spin that patch off as a bugfix PR, since presumably it affects all code not just vis?

In future, would appreciate a ref to this issue or to the supercsv bug thread in the bugfix commit as well as here.

@dhalperi

This comment has been minimized.

Show comment
Hide comment
@dhalperi

dhalperi Mar 10, 2014

Member

Also I believe that the code in #438 may have helped you debug this issue more easily -- it marks queries that fail as ERROR instead of KILLED and puts the "first" error reported to the master in the JSON encoding.

Member

dhalperi commented Mar 10, 2014

Also I believe that the code in #438 may have helped you debug this issue more easily -- it marks queries that fail as ERROR instead of KILLED and puts the "first" error reported to the master in the JSON encoding.

@dhalperi

This comment has been minimized.

Show comment
Hide comment
@dhalperi

dhalperi Mar 10, 2014

Member

Although, note that #438 was only tested for bugs when running a query rather than bugs when initializing/opening a query.

Member

dhalperi commented Mar 10, 2014

Although, note that #438 was only tested for bugs when running a query rather than bugs when initializing/opening a query.

@domoritz domoritz closed this Mar 11, 2014

@dhalperi

This comment has been minimized.

Show comment
Hide comment
@dhalperi

dhalperi Mar 11, 2014

Member

Just to addend: Dominik's initial request was "a tip for somewhere to look".

At the master, the exceptions sent back from every worker are logged to stderr. We looked in the master logs and saw that immediately before this error there was the CSV exception recorded above. This implied that the CSV exception was the actual root cause, and led us to the fix.

Member

dhalperi commented Mar 11, 2014

Just to addend: Dominik's initial request was "a tip for somewhere to look".

At the master, the exceptions sent back from every worker are logged to stderr. We looked in the master logs and saw that immediately before this error there was the CSV exception recorded above. This implied that the CSV exception was the actual root cause, and led us to the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment