Update Models for Filter Fields

[BradleySappington]

closes #1185

[BradleySappington]

closes #1186

[bhilbert4]

I was going to suggest getting these from constants.py, although I guess we may not always have all anomalies listed in constants.py in the models. So in that respect it's better to keep this separate from constants.py?

[BradleySappington]

@bhilbert4 , I have some shelved code that I'm going to add after this all gets merged. The Anomalies aren't done yet, i'm focusing on the header information for this merge (but wanted to at least get the anomalies in the models so we only have to do one migration).

[BradleySappington]

also! I'm still running a final test, I think the update will take many hours on my machine, and it will re-start if I lose any connectivity (it has happened once already), so there may be a small change or two that will come in if I find anything else
If you prefer I can tag you in a comment when I'm confident all is good

[melanieclarke]

@BradleySappington - can we please add expstart to the root file as well, since it's used for sorting? I thought obsstart was equivalent, but it isn't.

[BradleySappington]

@melanieclarke Sure can. I'll comment in here and remove the WIP tag when its ready for review

[BradleySappington]

@melanieclarke @bhilbert4 @mfixstsci - Ready for review.
Tested Successfully, jwql-test has up to date fits header information.

[melanieclarke]

This looks good!

I had some trouble getting django to pick up the migrations correctly in my local install. Since there wasn't anything of value in my local database, I just removed it and redid the migrations from the start. I am recreating the database now with archive_database_update and the new tables look good. I'll test some of my pending work against it when the update is complete.

One general question -- will the anomalies model replace the postgres table or be updated from it? And if updated, will it be by cron job or instantaneously when users make new selections? That class of data is a little different from the file info, since updates are made dynamically by JWQL users -- syncing sounds a little more challenging in this case.

[BradleySappington]

One general question -- will the anomalies model replace the postgres table or be updated from it? And if updated, will it be by cron job or instantaneously when users make new selections? That class of data is a little different from the file info, since updates are made dynamically by JWQL users -- syncing sounds a little more challenging in this case.

@melanieclarke
My plan is to create a script to migrate the postgres table into the django models. we would then update the django models directly from the submit anomaly form. and all anomaly information would be read direclty from the models. The postgres db would no longer be needed at all.

[BradleySappington]

When this gets merged all affected servers will need to:

1. run makemigrations
2. run migrate
3. temporarily comment out their cronjob that runs archive_database_update.py
4. run $python archive_database_update.py update
5. when complete re-enable removed cronjob

[melanieclarke]

My plan is to create a script to migrate the postgres table into the django models. we would then update the django models directly from the submit anomaly form. and all anomaly information would be read direclty from the models. The postgres db would no longer be needed at all.

Excellent, that sounds much cleaner.

[melanieclarke]

Everything looks good to me in local tests. It's taking a while to update my local database, but new tables look great for the instruments that have completed and I was able to find all the new fields I needed for the report download work.

[BradleySappington]

@melanieclarke The "update" flag will make this all take a very long time. Fortunately it shouldn't be too bad on our daily cronjobs because that flag will be omitted.