Support FORCE NOT NULL and FORCE NULL

[rdmurphy]

I've ran into some situations where django-postgres-copy is a little too simplistic about how it handles NULL fields. In some cases, I want an empty string to be just that — an empty string. (The best example of this is with CharField and TextField — I don't want them to be null=True and open up the possibility of multiple ways to have "no data.")

But sometimes I do want an empty string to become NULL, like with a DateField. The problem comes from when a single model is trying to account for both scenarios, and that's where the current null option in the command falls short.

Django is smart enough to smooth all this over in the ORM, but Postgres' COPY needs more guidance to make this possible. The way COPY FROM handles this is with the FORCE NOT NULL and FORCE NULL options. Both accept lists of column names.

FORCE NOT NULL

Do not match the specified columns' values against the null string. In the default case where the null string is empty, this means that empty values will be read as zero-length strings rather than nulls, even when they are not quoted. This option is allowed only in COPY FROM, and only when using CSV format.

FORCE NULL

Match the specified columns' values against the null string, even if it has been quoted, and if a match is found set the value to NULL. In the default case where the null string is empty, this converts a quoted empty string into NULL. This option is allowed only in COPY FROM, and only when using CSV format.

So in my case, an ideal result would be the ability to pass FORCE NOT NULL a list ((column_name [, ...])) of all the columns I want to allow an empty string to continue being an empty string.

There are a few ways to implement this. The most straightforward would be to add these two things as options, and leave the creating of the list of column names up to the user. You could get super fancy and do model field inspection to infer this in the command itself, but I can't fathom how many edge cases may come up.

Any thoughts?

[palewire]

Sounds great! I agree that the most straightforward way to go is allow users to pass in a list of column names. If you want to take a run at it, that would be great!