Small schema changes can cause large disturbances

[avikivity]

A schema change has an effect on the cache, on readers (since they need to perform a schema change) and perhaps many other components. Some changes are innocuous (like chancing the table comment) and yet cause a those large effects.

[avikivity]

This can be improved by splitting the schema into three nested components: primary_key_schema that describes only the primary key, and can never change, storage_schema, which adds non-primary-key columns to primary_key_schema, and schema, which adds everything else to storage_schema. We can then have each component use the smallest schema component it needs, for example comparators can use primary_key_schema while sstables, readers, and cache use storage_schema. Thus, they will see fewer changes from anciliary schema properties.

[slivne]

ref: #2718

[avikivity]

Closing as duplicate of #2718.