Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Could not parse subexpression ";" when parsing #109

Closed
dkam opened this issue Aug 23, 2014 · 20 comments
Closed

Error: Could not parse subexpression ";" when parsing #109

dkam opened this issue Aug 23, 2014 · 20 comments

Comments

@dkam
Copy link

dkam commented Aug 23, 2014

Hi there. I'm getting the following error when using the supplied deb on Ubuntu 14.04

dkam@localhost:~$ pgloader config.load 
STYLE-WARNING: Undefined alien: "SSLv2_client_method"
An unhandled error condition has been signalled:
   Could not parse subexpression ";" when parsing

 Expression PGLOADER.PARSER::COMMANDS
    Subexpression (+ PGLOADER.PARSER::COMMAND)
    Subexpression PGLOADER.PARSER::COMMAND
    Subexpression (AND
                   (OR PGLOADER.PARSER::LOAD-ARCHIVE
                       PGLOADER.PARSER::LOAD-CSV-FILE
                       PGLOADER.PARSER::LOAD-FIXED-COLS-FILE
                       PGLOADER.PARSER::LOAD-DBF-FILE
                       PGLOADER.PARSER::LOAD-MYSQL-DATABASE
                       PGLOADER.PARSER::LOAD-SQLITE-DATABASE
                       PGLOADER.PARSER::LOAD-SYSLOG-MESSAGES)
                   PGLOADER.PARSER::END-OF-COMMAND)
    Subexpression PGLOADER.PARSER::END-OF-COMMAND
    Subexpression (AND PGLOADER.PARSER::IGNORE-WHITESPACE #\;
                       PGLOADER.PARSER::IGNORE-WHITESPACE)
    Subexpression ";"

  Encountered at:
    LOAD DATABASE
FROM mysql
    ^ (Line 1, Column 0, Position 0)


; 
; compilation unit aborted
;   caught 1 fatal ERROR condition

My config is based on the sample:

LOAD DATABASE
        FROM mysql://user:password@host.com/dbname
        INTO postgresql://user:password@localhost/dbname
        INCLUDING ONLY TABLE NAMES MATCHING 'alerts'
        WITH include drop, create tables, create indexes, reset sequences;
        BEFORE LOAD DO  
        $$ create schema if not exists dbname; $$
@dimitri
Copy link
Owner

dimitri commented Aug 23, 2014

Your missing the ending semicolon after the last $$, and you have an extra one after reset sequences. For once the error message is kind of trustworthy here.

Note that the error messages situation is going to immensely improve as soon as the esrap issue about it is resolved, which is sooner by the day ;-) (nikodemus/esrap#26).

@dimitri dimitri closed this as completed Aug 23, 2014
@dkam
Copy link
Author

dkam commented Aug 24, 2014

Thanks dimitri!

I've removed the ; after reset sequences and added a ; following the $$ - however the error is still the same.

LOAD DATABASE
FROM mysql://user:pass@remotehost/dbname
INTO postgresql://user:pass@localhost/dbname
INCLUDING ONLY TABLE NAMES MATCHING 'alerts'
WITH include drop, create tables, create indexes, reset sequences
BEFORE LOAD DO  $$ create schema if not exists dbname; $$;

In fact, if I change the file to:

LOAD DATABASE
FROM mysql://user:pass@remotehost/dbname
INTO postgresql://user:pass@localhost/dbname
INCLUDING ONLY TABLE NAMES MATCHING 'alerts'
WITH include drop, create tables, create indexes, reset sequences;

I get the same error:

dkam@localhost:~$ pgloader config.load 
STYLE-WARNING: Undefined alien: "SSLv2_client_method"
An unhandled error condition has been signalled:
   Could not parse subexpression ";" when parsing

 Expression PGLOADER.PARSER::COMMANDS
    Subexpression (+ PGLOADER.PARSER::COMMAND)
    Subexpression PGLOADER.PARSER::COMMAND
    Subexpression (AND
                   (OR PGLOADER.PARSER::LOAD-ARCHIVE
                       PGLOADER.PARSER::LOAD-CSV-FILE
                       PGLOADER.PARSER::LOAD-FIXED-COLS-FILE
                       PGLOADER.PARSER::LOAD-DBF-FILE
                       PGLOADER.PARSER::LOAD-MYSQL-DATABASE
                       PGLOADER.PARSER::LOAD-SQLITE-DATABASE
                       PGLOADER.PARSER::LOAD-SYSLOG-MESSAGES)
                   PGLOADER.PARSER::END-OF-COMMAND)
    Subexpression PGLOADER.PARSER::END-OF-COMMAND
    Subexpression (AND PGLOADER.PARSER::IGNORE-WHITESPACE #\;
                       PGLOADER.PARSER::IGNORE-WHITESPACE)
    Subexpression ";"

  Encountered at:
    LOAD DATABASE 
FROM mysq
    ^ (Line 1, Column 0, Position 0)


; 
; compilation unit aborted
;   caught 1 fatal ERROR condition

@dkam
Copy link
Author

dkam commented Aug 24, 2014

If I strip the config down to:

LOAD DATABASE
FROM mysql://user:pass@remotehost.com/booko
INTO postgresql://user:pass@localhost/booko;

I get a different error:

dkam@localhost:~$ pgloader booko3.load 
STYLE-WARNING: Undefined alien: "SSLv2_client_method"
2014-08-24T14:43:44.061000+04:00 LOG Starting pgloader, log system is ready.
2014-08-24T14:43:44.101000+04:00 LOG Main logs in '/tmp/pgloader/pgloader.log'
2014-08-24T14:43:44.124000+04:00 LOG Data errors in '/tmp/pgloader/'
2014-08-24T14:43:44.124000+04:00 LOG Parsing commands from file #P"/home/dkam/booko3.load"
An unhandled error condition has been signalled:
   255 fell through ECASE expression. Wanted one of (10).

Perhaps something else is wrong with my setup?

@dimitri dimitri reopened this Aug 24, 2014
@dimitri
Copy link
Owner

dimitri commented Aug 25, 2014

Which version are you using when trying that? I can't reproduce the problem at all given current master's version, with neither of the syntaxes you've pasted here.

In older versions the ordering of the clauses did matter, so you could try placing the WITH clause before the INCLUDING ONLY one.

@dimitri dimitri closed this as completed Aug 25, 2014
@dkam
Copy link
Author

dkam commented Aug 25, 2014

Hi Dimitri - it's the version you've packaged as a deb. I've also tried master and the 3.0.99 tag. Could it be some other library / dependancy?

dkam@localhost:~$ /usr/bin/pgloader --version
STYLE-WARNING: Undefined alien: "SSLv2_client_method"
pgloader version "3.0.99"
compiled with SBCL 1.1.14.debian
dkam@localhost:~$ /usr/bin/pgloader booko3.load 
STYLE-WARNING: Undefined alien: "SSLv2_client_method"
2014-08-25T23:04:22.033000+04:00 LOG Starting pgloader, log system is ready.
2014-08-25T23:04:22.059000+04:00 LOG Main logs in '/tmp/pgloader/pgloader.log'
2014-08-25T23:04:22.068000+04:00 LOG Data errors in '/tmp/pgloader/'
2014-08-25T23:04:22.068000+04:00 LOG Parsing commands from file #P"/home/dkam/booko3.load"
An unhandled error condition has been signalled:
   255 fell through ECASE expression. Wanted one of (10).

@dimitri
Copy link
Owner

dimitri commented Aug 26, 2014

Did you try the line re-ordering trick, or building a newer version?

@dkam
Copy link
Author

dkam commented Aug 26, 2014

Currently, the config is simply:

LOAD DATABASE
FROM mysql://user:pass@remotehost.com/booko
INTO postgresql://user:pass@localhost/booko;

Building from the master branch, I get the following:

dkam@localhost:~/pgloader$ ./build/bin/pgloader ../booko3.load
2014-08-26T09:25:35.039000Z LOG Starting pgloader, log system is ready.
2014-08-26T09:25:35.065000Z LOG Main logs in '/tmp/pgloader/pgloader.log'
2014-08-26T09:25:35.075000Z LOG Data errors in '/tmp/pgloader/'
2014-08-26T09:25:35.075000Z LOG Parsing commands from file #P"/home/dkam/booko3.load"
An unhandled error condition has been signalled:
   255 fell through ECASE expression. Wanted one of (10).

During the build - these errors appear - could this be related?

;; loading file #P"/home/dkam/pgloader/src/hooks.lisp"
STYLE-WARNING: Undefined alien: "SSL_get_version"
STYLE-WARNING: Undefined alien: "SSL_load_error_strings"
STYLE-WARNING: Undefined alien: "SSL_library_init"
STYLE-WARNING: Undefined alien: "SSLv23_client_method"
STYLE-WARNING: Undefined alien: "SSLv23_server_method"
STYLE-WARNING: Undefined alien: "SSLv23_method"
STYLE-WARNING: Undefined alien: "SSLv3_client_method"
STYLE-WARNING: Undefined alien: "SSLv3_server_method"
STYLE-WARNING: Undefined alien: "SSLv3_method"
STYLE-WARNING: Undefined alien: "TLSv1_client_method"
STYLE-WARNING: Undefined alien: "TLSv1_server_method"
STYLE-WARNING: Undefined alien: "TLSv1_method"
STYLE-WARNING: Undefined alien: "SSL_CTX_new"
STYLE-WARNING: Undefined alien: "SSL_new"
STYLE-WARNING: Undefined alien: "SSL_get_fd"
STYLE-WARNING: Undefined alien: "SSL_set_fd"
STYLE-WARNING: Undefined alien: "SSL_set_bio"
STYLE-WARNING: Undefined alien: "SSL_get_error"
STYLE-WARNING: Undefined alien: "SSL_set_connect_state"
STYLE-WARNING: Undefined alien: "SSL_set_accept_state"
STYLE-WARNING: Undefined alien: "SSL_connect"
STYLE-WARNING: Undefined alien: "SSL_accept"
STYLE-WARNING: Undefined alien: "SSL_write"
STYLE-WARNING: Undefined alien: "SSL_read"
STYLE-WARNING: Undefined alien: "SSL_shutdown"
STYLE-WARNING: Undefined alien: "SSL_free"
STYLE-WARNING: Undefined alien: "SSL_CTX_free"
STYLE-WARNING: Undefined alien: "BIO_ctrl"
STYLE-WARNING: Undefined alien: "BIO_new_socket"
STYLE-WARNING: Undefined alien: "BIO_new"
STYLE-WARNING: Undefined alien: "ERR_get_error"
STYLE-WARNING: Undefined alien: "ERR_error_string"
STYLE-WARNING: Undefined alien: "SSL_set_cipher_list"
STYLE-WARNING: Undefined alien: "SSL_use_RSAPrivateKey_file"
STYLE-WARNING: Undefined alien: "SSL_CTX_use_RSAPrivateKey_file"
STYLE-WARNING: Undefined alien: "SSL_use_certificate_file"
STYLE-WARNING: Undefined alien: "SSL_CTX_use_certificate_chain_file"
STYLE-WARNING: Undefined alien: "SSL_CTX_load_verify_locations"
STYLE-WARNING: Undefined alien: "SSL_CTX_set_client_CA_list"
STYLE-WARNING: Undefined alien: "SSL_load_client_CA_file"
STYLE-WARNING: Undefined alien: "SSL_CTX_ctrl"
STYLE-WARNING: Undefined alien: "SSL_CTX_set_default_passwd_cb"
STYLE-WARNING: Undefined alien: "CRYPTO_num_locks"
STYLE-WARNING: Undefined alien: "CRYPTO_set_locking_callback"
STYLE-WARNING: Undefined alien: "CRYPTO_set_id_callback"
STYLE-WARNING: Undefined alien: "RAND_seed"
STYLE-WARNING: Undefined alien: "RAND_bytes"
STYLE-WARNING: Undefined alien: "SSL_CTX_set_verify_depth"
STYLE-WARNING: Undefined alien: "SSL_get_verify_result"
STYLE-WARNING: Undefined alien: "SSL_get_peer_certificate"
STYLE-WARNING: Undefined alien: "X509_free"
STYLE-WARNING: Undefined alien: "X509_NAME_oneline"
STYLE-WARNING: Undefined alien: "X509_get_issuer_name"
STYLE-WARNING: Undefined alien: "X509_get_subject_name"
STYLE-WARNING: Undefined alien: "SSL_CTX_set_default_verify_paths"
STYLE-WARNING: Undefined alien: "RSA_generate_key"
STYLE-WARNING: Undefined alien: "RSA_free"
STYLE-WARNING: Undefined alien: "SSL_CTX_set_tmp_rsa_callback"
[undoing binding stack and other enclosing state... done]

@dimitri
Copy link
Owner

dimitri commented Aug 26, 2014

Oh, ok, so that's not a parsing problem. I think it's https://github.com/qitab/qmynd/blob/master/src/mysql-protocol/handshake.lisp#l209 wherein we only support the MySQL protocol from 4.1 onward.

What version of MySQL is this server running?

@dimitri dimitri reopened this Aug 26, 2014
@dkam
Copy link
Author

dkam commented Aug 26, 2014

It's running 5.5: Server version: 5.5.38-0ubuntu0.12.04.1-log

@dimitri
Copy link
Owner

dimitri commented Aug 27, 2014

Ok can you try with the --debug so that we have a backtrace?

@dkam
Copy link
Author

dkam commented Aug 27, 2014

Sure thing!

dkam@localhost:~/pgloader$ ./build/bin/pgloader --debug ../booko3.load
sb-impl::*default-external-format* :UTF-8
tmpdir: #P"/tmp/pgloader/"
2014-08-27T06:59:22.038000Z LOG Starting pgloader, log system is ready.
2014-08-27T06:59:22.063000Z INFO Starting monitor
2014-08-27T06:59:22.073000Z LOG Main logs in '/tmp/pgloader/pgloader.log'
2014-08-27T06:59:22.073000Z LOG Data errors in '/tmp/pgloader/'
2014-08-27T06:59:22.073000Z LOG Parsing commands from file #P"/home/dkam/booko3.load"
2014-08-27T06:59:22.073000Z DEBUG CONNECT
2014-08-27T06:59:22.073000Z DEBUG BEGIN
An unhandled error condition has been signalled:
   255 fell through ECASE expression. Wanted one of (10).

Date/time: 2014-08-27-06:59An unhandled error condition has been signalled:
                              255 fell through ECASE expression.
                              Wanted one of (10).


Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1006983573}>
0: ((LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX))
1: (SB-IMPL::CALL-WITH-SANE-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {1007984D5B}>)
2: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {1007984D2B}>)
3: (PRINT-BACKTRACE :STREAM #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {10002BBC23}> :START 0 :FROM :DEBUGGER-FRAME :COUNT 4611686018427387903 :PRINT-THREAD T :PRINT-FRAME-SOURCE NIL :METHOD-FRAME-STYLE NIL)
4: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {10002BBC23}>)
5: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE #<SB-KERNEL:CASE-FAILURE expected-type: (MEMBER 10) datum: 255> :OUTPUT #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {10002BBC23}> :IF-EXISTS :APPEND :VERBOSE T)
6: (SIGNAL #<SB-KERNEL:CASE-FAILURE expected-type: (MEMBER 10) datum: 255>)
7: (ERROR SB-KERNEL:CASE-FAILURE :NAME ECASE :DATUM 255 :EXPECTED-TYPE (MEMBER 10) :POSSIBILITIES (10))
8: (SB-KERNEL:CASE-FAILURE ECASE 255 (10))
9: (QMYND-IMPL:PROCESS-INITIAL-HANDSHAKE-PAYLOAD #<QMYND-IMPL::MY-PACKET-STREAM 0/78 [1]>)
10: (QMYND-IMPL::MYSQL-CONNECT-DO-HANDSHAKE #<QMYND-IMPL::MYSQL-INET-CONNECTION {100797F393}> "user" "password" "database" :COMPRESS NIL :SSL :UNSPECIFIED :SSL-VERIFY NIL)
11: (PGLOADER.MYSQL::FETCH-MYSQL-METADATA :STATE #S(PGLOADER.UTILS::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 1 {1006B8EDF3}> :TABNAMES ("fetch meta data") :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0) :MATERIALIZE-VIEWS NIL :ONLY-TABLES NIL :INCLUDING NIL :EXCLUDING NIL)
12: ((:METHOD PGLOADER.SOURCES:COPY-DATABASE (PGLOADER.MYSQL:COPY-MYSQL)) #<PGLOADER.MYSQL:COPY-MYSQL {10075F5E23}> :STATE-BEFORE #S(PGLOADER.UTILS::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 1 {1006B8EDF3}> :TABNAMES ("fetch meta data") :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0) :STATE-AFTER #S(PGLOADER.UTILS::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {1006B8FB73}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0) :STATE-INDEXES #S(PGLOADER.UTILS::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {1006B8F6F3}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0) :TRUNCATE NIL :DATA-ONLY NIL :SCHEMA-ONLY NIL :CREATE-TABLES NIL :INCLUDE-DROP NIL :CREATE-INDEXES NIL :RESET-SEQUENCES NIL :FOREIGN-KEYS NIL :IDENTIFIER-CASE :DOWNCASE :ONLY-TABLES NIL :INCLUDING NIL :EXCLUDING NIL :DECODING-AS NIL :MATERIALIZE-VIEWS NIL) [fast-method]
13: ((LAMBDA ()))
14: (PGLOADER.PARSER:RUN-COMMANDS #P"/home/dkam/booko3.load" :START-LOGGER NIL :SUMMARY NIL :LOG-FILENAME NIL :LOG-MIN-MESSAGES NIL :CLIENT-MIN-MESSAGES NIL)
15: ((FLET #:CLEANUP-FUN-200 :IN PGLOADER::MAIN)) [cleanup]
16: (PGLOADER::MAIN ("./build/bin/pgloader" "--debug" "../booko3.load"))
17: ((LAMBDA NIL :IN "/home/dkam/pgloader/dumper-2SKVI5f7.lisp"))
18: ((FLET #:WITHOUT-INTERRUPTS-BODY-54 :IN SAVE-LISP-AND-DIE))
19: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))


debugger invoked on a SB-KERNEL:CASE-FAILURE in thread
#<THREAD "main thread" RUNNING {1006983573}>:
  255 fell through ECASE expression. Wanted one of (10).

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit application


debugger invoked on a TYPE-ERROR in thread
#<THREAD "main thread" RUNNING {1006983573}>:
  The value INVOKE-DEBUGGER is not of type SB-DI:FRAME.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit application

(SB-DEBUG::FRAME-CALL INVOKE-DEBUGGER :METHOD-FRAME-STYLE :NORMAL :REPLACE-DYNAMIC-EXTENT-OBJECTS NIL)
0[2] 2014-08-27T06:59:22.274000Z FATAL We have a situation here.
dkam@localhost:~/pgloader$ 

@dimitri
Copy link
Owner

dimitri commented Aug 27, 2014

Is there a configuration knob in MySQL so that it would run with the old protocol? My reading of http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::Handshake makes it a compile-time option, and I don't know how you could check that...

@dkam
Copy link
Author

dkam commented Sep 7, 2014

Hey Dimitri,
I'm not sure if it's possible to do and since these hosts are in a master / slave setup, I'm not keen on changing the protocol version. Is there something else I could try?

Thanks,
Dan

@dimitri
Copy link
Owner

dimitri commented Sep 8, 2014

Can you tell me which version exactly you're using? It might be that they have a different protocol that the stock MySQL one...

@dkam
Copy link
Author

dkam commented Sep 9, 2014

Mysql: Server version: 5.5.38-0ubuntu0.12.04.1-log (Ubuntu)
OS: Ubuntu 12.04.5 LTS

@dimitri
Copy link
Owner

dimitri commented Sep 9, 2014

I just improved the situation just a little bit here: qitab/qmynd@fc1b3db

Now when using the current version of the driver you should have a proper error message. To update the driver, you can do:

git clone https://github.com/qitab/qmynd.git ./build/quicklisp/local-projects/qmynd

Then you have to rebuild pgloader. Note that make clean will remove the special version of the driver and you will have to git clone again to reproduce your build.

Also I'm hinted in http://bugs.mysql.com/bug.php?id=64870 that it might be an SSL problem, can you try without SSL if you're actually already using it?

@dimitri
Copy link
Owner

dimitri commented Oct 5, 2014

Any news?

@dkam
Copy link
Author

dkam commented Oct 6, 2014

I've pulled the latest changes in master, cloned the qmynd repo and built it (within the pgload directory ), then build pgloader.

dkam@localhost:~/pgloader$ ./build/bin/pgloader ../booko2.load 
2014-10-06T01:03:11.035000Z LOG Starting pgloader, log system is ready.
2014-10-06T01:03:11.058000Z LOG Main logs in 'NIL'
2014-10-06T01:03:11.071000Z LOG Data errors in '//tmp/pgloader//'
2014-10-06T01:03:11.071000Z LOG Parsing commands from file #P"/home/dkam/booko2.load"
An unhandled error condition has been signalled:
   Condition QMYND-IMPL::PROTOCOL-VERSION-MISMATCH was signalled.

I believe this is what you were expecting? But now with a clearer error. I'll try without SSL now.

The remote mysql server is : Server version: 5.5.38-0ubuntu0.12.04.1-log (Ubuntu)

@dimitri
Copy link
Owner

dimitri commented Oct 6, 2014

Thanks for retrying, this is more in line with normal users expectations, despite still failing. At least we get a grasp why directly from the error message, without having to look at the source code...

From there, I don't know exactly what to do, your MySQL server isn't using the Protocol Version 10 apparently, at that's the only one known by the driver, and the only other one documented that I found is Protocol 9 (MySQL 3.x era), certainly not 255 as we saw before in your logs.

@dimitri
Copy link
Owner

dimitri commented Jan 14, 2015

I'm closing that one to clean up the number of open issues and for lack of new elements allowing to make progress. If you have some, please consider re-opening with them, of course.

@dimitri dimitri closed this as completed Jan 14, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants