Doc: remove the statement of SET NOT NULL from the spark-ddl doc, it is not supported.

[zhongyujiang]

No description provided.

[rdblue]

@zhongyujiang, it sounds like this is a regression, not a documentation problem do you know what the issue is?

@szehon-ho, any ideas?

[zhongyujiang]

@rdblue I think it's just a doc bug, SET NOT NULL was not supported before either due to restrictions from Spark. There is a test for this actually:

iceberg/spark/v3.2/spark/src/test/java/org/apache/iceberg/spark/sql/TestAlterTable.java

Lines 203 to 217 in bb18f18

	public void testAlterColumnSetNotNull() {
	// no-op changes are allowed
	sql("ALTER TABLE %s ALTER COLUMN id SET NOT NULL", tableName);
	Types.StructType expectedSchema = Types.StructType.of(
	NestedField.required(1, "id", Types.LongType.get()),
	NestedField.optional(2, "data", Types.StringType.get()));
	Assert.assertEquals("Schema should match expected",
	expectedSchema, validationCatalog.loadTable(tableIdent).schema().asStruct());
	AssertHelpers.assertThrows("Should reject adding NOT NULL constraint to an optional column",
	AnalysisException.class, "Cannot change nullable column to non-nullable: data",
	() -> sql("ALTER TABLE %s ALTER COLUMN data SET NOT NULL", tableName));
	}

[rdblue]

I see. The problem is that there isn't a way to override the check from SQL. Iceberg does allow you to make incompatible changes if you decide to "force" the change. But the SQL API doesn't have a way to do that an will always fail.

Looks like Postgres supports this by immediately checking the table to see if there are any null values. We should be able to use a metadata check to do the same, assuming that the column's metadata collection is not set to none.

[szehon-ho]

Yea, that sounds like a cool feature to have

[zhongyujiang]

+1 Sounds like a cool feature!

[zhongyujiang]

Hey @szehon-ho @rdblue, could you please help review this when you have time? Thanks. (I've seen users misled by this)

[zhongyujiang]

Looks like Postgres supports this by immediately checking the table to see if there are any null values. We should be able to use a metadata check to do the same, assuming that the column's metadata collection is not set to none.

This is cool and I'd like to give a try on this.
I did some investigation, I think we may need to check the metadata of all files (including deleted data file), because CDC read workload may read deleted files, we also need to check the fields in the equality delete file, because the equality fields are derived from the table schema. This ensures that not-null constraint updates are totally compatible. What do you think ? @szehon-ho @rdblue

[zhongyujiang]

Close this as the doc has been fixed by another PR.