Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compaction_manager - compaction failed: sstables::malformed_sstable_exception (end of input, but not end of partition) #4533

Closed
haaawk opened this issue Jun 11, 2019 · 9 comments

Comments

Projects
None yet
4 participants
@haaawk
Copy link
Member

commented Jun 11, 2019

Issue reported by @r4fek.

report_id: db8df7c6-43f9-4ce8-af1a-214c38c7051c
Scylla version: 3.0.6 and 3.0.7

cze 04 06:58:43 ams-dbs10 scylla[18757]:  [shard 13] compaction - Compacting [/var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-10595193-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-4737047-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-5255237-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1152543-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1336635-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-634629-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-990991-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-828749-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1546579-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-415807-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-6768039-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-122787-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-166901-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-474641-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-361527-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1781225-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-7052135-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-602613-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-930823-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-2370669-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-3204649-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1915821-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-3368639-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-777735-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-534303-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-2049129-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-2221675-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-7333839-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-262949-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1679059-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-6563155-big-Data.db:level=0, /var/lib/scylla/data/sync/user_stats-2636d56072f211e8809a00000000001e/mc-1459179-big-Data.db:level=0, ]
cze 04 06:58:44 ams-dbs10 scylla[18757]:  [shard 13] compaction_manager - compaction failed: sstables::malformed_sstable_exception (end of input, but not end of partition)

schema:

CREATE TABLE sync.user_stats (
    user_id text PRIMARY KEY,
    clients_usage map<text, timestamp>,
    last_seen timestamp
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'LeveledCompactionStrategy', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 432000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';
@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

The problem seems to be with mc-122787-big-Data.db

Exception in thread "main" org.apache.cassandra.serializers.MarshalException: String didn't validate.
	at org.apache.cassandra.serializers.UTF8Serializer.validate(UTF8Serializer.java:35)
	at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:131)
	at org.apache.cassandra.tools.JsonTransformer.serializeCell(JsonTransformer.java:429)
	at org.apache.cassandra.tools.JsonTransformer.serializeColumnData(JsonTransformer.java:405)
	at org.apache.cassandra.tools.JsonTransformer.serializeRow(JsonTransformer.java:280)
	at org.apache.cassandra.tools.JsonTransformer.serializePartition(JsonTransformer.java:211)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.apache.cassandra.tools.JsonTransformer.toJson(JsonTransformer.java:101)
	at org.apache.cassandra.tools.SSTableExport.run(SSTableExport.java:248)
	at com.scylladb.tools.SSTableExport.main(SSTableExport.java:44)

in following part:

{
    "partition" : {
      "key" : [ "457570552" ],
      "position" : 102630
    },
    "rows" : [
      {
        "type" : "row",
        "position" : 102688,
        "cells" : [
          { "name" : "last_seen", "value" : "2019-05-05 00:37:07.130Z", "tstamp" : "2019-05-05T00:37:07.131321Z" },
          { "name" : "clients_usage", "path" : [ ] } ] } ] } ]
@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

The problem seems to be a wrong value of a key in a map entry.

BTW @r4fek why did you decide to use a map instead of modelling the data like this:

CREATE TABLE sync.user_stats (
    user_id text,
    device_id text,
    last_seen timestamp,
    PRIMARY KEY(user_id, device_id)
)

The global last_seen for user would be a very cheap query select min(last_seen) from user_stats where user_id = ?. Assuming users don't use tens or hundreds of devices.

@r4fek

This comment has been minimized.

Copy link

commented Jun 11, 2019

Yeah, it is implemented like that due some historical reasons. Can we do something with this wrong key value? How did it get there?

@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

I'm still investigating. Will get back to you.

@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2019

Here's the value (in bytes) of key that creates a problem: [0, 0, 1, 106, -123, 108, 71, -70]

@slivne slivne added this to the 3.1 milestone Jun 13, 2019

@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 13, 2019

Unfortunately @r4fek has only logs since 30 May and the problematic sstable was created on 5 May so it will be difficult to say whether the sstable was created by compaction or memtable flush.

@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2019

It turns out that there is a bug on a write path when writing maps to sstables. If a key has an empty value then it's not written to sstable at all which is wrong. Those values are serialized to length + data so empty key should write length = 0 and no data. That's what's expected by the read path. I have a fix that I will send for review soon.

@slivne

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

@haaawk - is this confined to MC format ?

I want to validate that this issue does not happen with KA/LA formats

@haaawk

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2019

@slivne the bug is in MC writer. LA/KA has different writer which seems to be ok but I haven't tested it.

avikivity added a commit that referenced this issue Jun 16, 2019

sstables: distinguish empty and missing cellpath
Before this patch mc sstables writer was ignoring
empty cellpaths. This is a wrong behaviour because
it is possible to have empty key in a map. In such case,
our writer creats a wrong sstable that we can't read back.
This is becaus a complex cell expects cellpath for each
simple cell it has. When writer ignores empty cellpath
it writes nothing and instead it should write a length
of zero to the file so that we know there's an empty cellpath.

Fixes #4533

Tests: unit(release)

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <46242906c691a56a915ca5994b36baf87ee633b7.1560532790.git.piotr@scylladb.com>
(cherry picked from commit a41c976)

avikivity added a commit that referenced this issue Jun 16, 2019

sstables: distinguish empty and missing cellpath
Before this patch mc sstables writer was ignoring
empty cellpaths. This is a wrong behaviour because
it is possible to have empty key in a map. In such case,
our writer creats a wrong sstable that we can't read back.
This is becaus a complex cell expects cellpath for each
simple cell it has. When writer ignores empty cellpath
it writes nothing and instead it should write a length
of zero to the file so that we know there's an empty cellpath.

Fixes #4533

Tests: unit(release)

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <46242906c691a56a915ca5994b36baf87ee633b7.1560532790.git.piotr@scylladb.com>
(cherry picked from commit a41c976)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.