Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker out-of-data-space bug #31

Closed
roadscape opened this issue Nov 17, 2017 · 1 comment
Closed

docker out-of-data-space bug #31

roadscape opened this issue Nov 17, 2017 · 1 comment
Labels
priority/low
Milestone
1.0

Comments

@roadscape
Copy link
Contributor

This happens sometimes. Possibly due to db error log?

Oct 11 08:14:11 docker/cd60a907a168[7347]: [SYNC] Got block 16231286 (68.7/s, 1096rps 73wps) -- 0.0m remaining
Oct 11 08:14:11 docker/cd60a907a168[7347]: [INIT] *** Initial sync complete. Rebuilding cache. ***
Oct 11 08:14:11 docker/cd60a907a168[7347]: [INIT] Found 15034490 missing post cache entries
Oct 11 08:14:56 kernel: [1692646.139464] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode
Oct 11 08:14:56 kernel: [1692646.144088] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2519552)
Oct 11 08:14:56 kernel: [1692646.150472] Buffer I/O error on device dm-4, logical block 2519552
Oct 11 08:14:56 kernel: [1692646.153929] Buffer I/O error on device dm-4, logical block 2519553
Oct 11 08:14:56 kernel: [1692646.157063] Buffer I/O error on device dm-4, logical block 2519554
Oct 11 08:14:56 kernel: [1692646.160150] Buffer I/O error on device dm-4, logical block 2519555
Oct 11 08:14:56 kernel: [1692646.163072] Buffer I/O error on device dm-4, logical block 2519556
Oct 11 08:14:56 kernel: [1692646.165959] Buffer I/O error on device dm-4, logical block 2519557
Oct 11 08:14:56 kernel: [1692646.168900] Buffer I/O error on device dm-4, logical block 2519558
Oct 11 08:14:56 kernel: [1692646.171721] Buffer I/O error on device dm-4, logical block 2519559
Oct 11 08:14:56 kernel: [1692646.174681] Buffer I/O error on device dm-4, logical block 2519560
Oct 11 08:14:56 kernel: [1692646.177909] Buffer I/O error on device dm-4, logical block 2519561
Oct 11 08:14:56 kernel: [1692646.181019] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2519808)
Oct 11 08:14:56 kernel: [1692646.187836] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520064)
Oct 11 08:14:56 kernel: [1692646.194356] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520320)
Oct 11 08:14:56 kernel: [1692646.200836] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520576)
Oct 11 08:14:56 kernel: [1692646.207192] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1048576000 size 8388608 starting block 2520832)
Oct 11 08:14:56 kernel: [1692646.214593] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521088)
Oct 11 08:14:56 kernel: [1692646.221018] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521344)
Oct 11 08:14:56 kernel: [1692646.227704] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521600)
Oct 11 08:14:56 kernel: [1692646.234261] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1056964608 size 8388608 starting block 2521856)
Oct 11 08:14:56 kernel: [1692646.271736] JBD2: Detected IO errors while flushing file data on dm-4-8
Oct 11 08:15:05 kernel: [1692655.338808] EXT4-fs warning: 287 callbacks suppressed
Oct 11 08:15:05 kernel: [1692655.342246] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 0 size 0 starting block 2519552)
Oct 11 08:15:05 kernel: [1692655.351639] buffer_io_error: 75798 callbacks suppressed
Oct 11 08:15:05 kernel: [1692655.354860] Buffer I/O error on device dm-4, logical block 2519552
Oct 11 08:15:05 kernel: [1692655.358579] Buffer I/O error on device dm-4, logical block 2519553
Oct 11 08:15:05 kernel: [1692655.362446] Buffer I/O error on device dm-4, logical block 2519554
Oct 11 08:15:05 kernel: [1692655.366161] Buffer I/O error on device dm-4, logical block 2519555
Oct 11 08:15:05 kernel: [1692655.370238] Buffer I/O error on device dm-4, logical block 2519556
Oct 11 08:15:05 kernel: [1692655.373862] Buffer I/O error on device dm-4, logical block 2519557
Oct 11 08:15:05 kernel: [1692655.377500] Buffer I/O error on device dm-4, logical block 2519558
Oct 11 08:15:05 kernel: [1692655.381224] Buffer I/O error on device dm-4, logical block 2519559
Oct 11 08:15:05 kernel: [1692655.384238] Buffer I/O error on device dm-4, logical block 2519560
Oct 11 08:15:05 kernel: [1692655.387449] Buffer I/O error on device dm-4, logical block 2519561
Oct 11 08:15:05 kernel: [1692655.390647] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2519808)
Oct 11 08:15:05 kernel: [1692655.402506] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520064)
Oct 11 08:15:05 kernel: [1692655.411143] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520320)
Oct 11 08:15:05 kernel: [1692655.420756] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520576)
Oct 11 08:15:05 kernel: [1692655.429972] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2520832)
Oct 11 08:15:05 kernel: [1692655.439005] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521088)
Oct 11 08:15:05 kernel: [1692655.448575] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521344)
Oct 11 08:15:05 kernel: [1692655.457532] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521600)
Oct 11 08:15:05 kernel: [1692655.466855] EXT4-fs warning (device dm-4): ext4_end_bio:314: I/O error -28 writing to inode 528049 (offset 1361182720 size 6160384 starting block 2521856)
Oct 11 08:15:06 kernel: [1692655.516649] JBD2: Detected IO errors while flushing file data on dm-4-8
Oct 11 08:15:09 docker/cd60a907a168[7347]: #033[93m[SQL][58385ms] SELECT id, author, permlink FROM hive_posts WHERE is_deleted = 0 AND id > (SELECT IFNULL(MAX(post_id), 0) FROM hive_posts_cache) ORDER BY id LIMIT 1000000#033[0m
Oct 11 08:15:15 docker/cd60a907a168[7347]: Traceback (most recent call last):
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
Oct 11 08:15:15 docker/cd60a907a168[7347]:     context)
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
Oct 11 08:15:15 docker/cd60a907a168[7347]:     cursor.execute(statement, parameters)
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 250, in execute
Oct 11 08:15:15 docker/cd60a907a168[7347]:     self.errorhandler(self, exc, value)
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
Oct 11 08:15:15 docker/cd60a907a168[7347]:     raise errorvalue
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 247, in execute
Oct 11 08:15:15 docker/cd60a907a168[7347]:     res = self._query(query)
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 411, in _query
Oct 11 08:15:15 docker/cd60a907a168[7347]:     rowcount = self._do_query(q)
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/MySQLdb/cursors.py", line 374, in _do_query
Oct 11 08:15:15 docker/cd60a907a168[7347]:     db.query(q)
Oct 11 08:15:15 docker/cd60a907a168[7347]:   File "/usr/local/lib/python3.5/dist-packages/MySQLdb/connections.py", line 277, in query
Oct 11 08:15:15 docker/cd60a907a168[7347]:     _mysql.connection.query(self, query)
Oct 11 08:15:15 docker/cd60a907a168[7347]: _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1")
Oct 11 08:15:15 docker/cd60a907a168[7347]:
Oct 11 08:15:15 docker/cd60a907a168[7347]: The above exception was the direct cause of the following exception:
Oct 11 08:15:15 docker/cd60a907a168[7347]:
@roadscape roadscape added this to the 1.0 milestone Nov 17, 2017
@roadscape
Copy link
Contributor Author

will be resolved with #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/low
Projects
None yet
Milestone
1.0
Development

No branches or pull requests
1 participant
@roadscape