Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

errors while replicating #57

Closed
vadimtk opened this issue Dec 25, 2017 · 1 comment
Closed

errors while replicating #57

vadimtk opened this issue Dec 25, 2017 · 1 comment

Comments

@vadimtk
Copy link

vadimtk commented Dec 25, 2017

I am trying to replication the table from MySQL

CREATE TABLE `acct` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `stamp_inserted` datetime NOT NULL,
  `ip_src` char(45) NOT NULL,
  `ip_dst` char(45) NOT NULL,
  `src_port` int(2) unsigned NOT NULL,
  `dst_port` int(2) unsigned NOT NULL,
  `in_bytes` int(10) unsigned NOT NULL,
  `out_bytes` int(10) unsigned NOT NULL,
  `src_host` varchar(1024) DEFAULT NULL,
  `dst_host` varchar(1024) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1391121 DEFAULT CHARSET=latin1

to a ClickHouse table

CREATE TABLE default.acct ( id UInt32,  stamp_inserted DateTime,  ip_src String,  ip_dst String,  src_port UInt16,  dst_port UInt16,  in_bytes UInt32,  out_bytes UInt32,  src_host String,  dst_host String) ENGINE = MergeTree PARTITION BY toMonday(stamp_inserted) ORDER BY (stamp_inserted, ip_dst) SETTINGS index_granularity = 8192

my script is

python3 -m clickhouse_mysql.main \
    --src-resume \
    --src-wait \
    --nice-pause=1 \
    --log-level=info \
    --log-file=acct.log \
    --src-host=127.0.0.1 \
    --src-user=reader \
    --src-password=qwerty \
    --dst-host=127.0.0.1 \
    --csvpool \
    --csvpool-file-path-prefix=qwe_ \
    --mempool-max-flush-interval=60 \
    --mempool-max-events-num=10000

while it is running and the events are coming to MySQL, the script fails as:

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/processwriter.py", line 30, in process   
    writer.insert(event_or_events)
  File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/csvwriter.py", line 83, in insert
    self.fieldnames = sorted(events[0].column_names())
  File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/event/event.py", line 53, in column_names
    return self.mysql_event.rows[0]['values'].keys()
IndexError: list index out of range
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/processwriter.py", line 30, in process
    writer.insert(event_or_events)
  File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/writer/csvwriter.py", line 83, in insert
    self.fieldnames = sorted(events[0].column_names())
  File "/root/clickhousr/click-mysql/clickhouse-mysql-data-reader/clickhouse_mysql/event/event.py", line 53, in column_names
    return self.mysql_event.rows[0]['values'].keys()
IndexError: list index out of range

the log is

2017-12-25 13:15:31,936/1514236531.936720:INFO:Starting
2017-12-25 13:15:31,940/1514236531.940504:INFO:raw dbs list len=0
2017-12-25 13:15:31,941/1514236531.941264:INFO:dbs list len=0
2017-12-25 13:15:31,941/1514236531.941383:INFO:raw tables list len=0
2017-12-25 13:15:31,941/1514236531.941494:INFO:tables list len=0
2017-12-25 13:16:24,682/1514236584.682820:INFO:rot now:1514236584.682676 bktttl:1 bktitemsttl: 13 index:t1.acct reason:FLUSH bktsonbelt:2 bktsize:13 beltnum:1
2017-12-25 13:16:44,976/1514236604.976741:INFO:rot now:1514236604.976535 bktttl:2 bktitemsttl: 52 index:t1.acct reason:FLUSH bktsonbelt:2 bktsize:39 beltnum:1
2017-12-25 13:16:44,980/1514236604.980124:INFO:PERF - buckets_per_sec:0.049276 items_per_sec:1.921764 for last 20 sec
@sunsingerus
Copy link
Collaborator

New version is published. Please, take a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants