Further instructions on how to setup iop4

[morcuended]

Still not clear to me how to set the configuration file (path to real DB, astrometry cache, etc).

I guess related it's related to knowing if these two scripts actually work

    $ python manage.py makemigrations
    $ python manage.py migrate

The first one returns No changes detected

The second one seems to run fine (Running migration messages and so on)

Then, the command python manage.py dumpdata --natural-primary --natural-foreign --format=yaml > priv.dumps.yaml returns:

CommandError: Unable to serialize database: no such table: iop4api_astrosource

Also the test test_build_reduced_multiproc takes too long, I guess because it's trying to download astrometry files.

[juanep97]

Also the test test_build_reduced_multiproc takes too long, I guess because it's trying to download astrometry files.

Yes, it is tens of GB, it will take quite a bit to do the initial download.

Then, the command python manage.py dumpdata --natural-primary --natural-foreign --format=yaml > priv.dumps.yaml returns:

CommandError: Unable to serialize database: no such table: iop4api_astrosource

That should not happen, did you add the db_path variable in config.yaml? Can you check if the file it points to already exists and report back? It should have been created with the migrations commands.

[morcuended]

Then, the command python manage.py dumpdata --natural-primary --natural-foreign --format=yaml > priv.dumps.yaml returns:

CommandError: Unable to serialize database: no such table: iop4api_astrosource
That should not happen, did you add the db_path variable in config.yaml? Can you check if the file it points to already exists and report back? It should have been created with the migrations commands.

db_path is defined in the config file, the db file is created after the first two commands: python manage.py makemigrations and python manage.py migrate. Next step is to populate it with default data (catalog) from a yaml file priv.dumps.yaml that I don't have. I thought that this dump yml file was created with the next command:

    $ python manage.py dumpdata --natural-primary --natural-foreign --format=yaml > priv.dumps.yaml

But running this gives me: CommandError: Unable to serialize database: no such table: iop4api_astrosource

Is this the correct sequence to set it up? Do I need the catalog file to populate it?

[morcuended]

Also, when I run the test test_build_reduced_multiproc, it just seems stuck. It was not downloading anything to the astrometry cache dir I defined. I downloaded some astrometry index files by hand and still the test was taking a lot of time and never finishing.

I tried to run the test interactively, and doing from iop4lib.db import Epoch was giving me:

ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

I previously set up the config file as:

from tests.conftest import TEST_CONFIG

import iop4lib.config

iop4conf = iop4lib.Config(config_path=TEST_CONFIG)

[juanep97]

If it started downloading when trying to do it manually but not when running the tests, it might be related to pytest?

You could try running the tests showing standard output and logging (pytest -vxs o log_cli=true), and commenting the line
root.addFilter(logging.Filter("iop4lib")) in parallel.py. You should see the astrometry module attempting to connect and download files.

Could you check this?

Also, you may try running with max_concurrent_threads=1 in the config, to discard problems with multiprocessing.

[morcuended]

I set max_concurrent_threads=1 but I still have the same problem. The last test keeps hanging forever, even if it says Aborted. See the output below:

(this is with version of PR #21)

tests/test_osnt090.py::test_testdata Creating test database for alias 'default' ('/Users/dmorcuende/iop4/test_iop4.db')...
Destroying old test database for alias 'default' ('/Users/dmorcuende/iop4/test_iop4.db')...
Operations to perform:
  Synchronize unmigrated apps: iop4api
  Apply all migrations: (none)
Synchronizing apps without migrations:
  Creating tables...
    Creating table iop4api_astrosource
    Creating table iop4api_epoch
    Creating table iop4api_rawfit
    Creating table iop4api_masterbias
    Creating table iop4api_masterflat
    Creating table iop4api_photopolresultreducedfitrelation
    Creating table iop4api_photopolresult
    Creating table iop4api_aperphotresult
    Creating table iop4api_reducedfit
    Running deferred SQL...
Running migrations:
  No migrations to apply.
PASSED
tests/test_osnt090.py::test_testconfig_testdb PASSED
tests/test_osnt090.py::test_epoch_creation PASSED
tests/test_osnt090.py::test_epoch_masterbias_masterflats PASSED
tests/test_osnt090.py::test_build_single_proc objc[72782]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[72782]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
Fatal Python error: Aborted

[morcuended]

it seems that by doing exporting first export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES at least now it is downloading astrometry files

[juanep97]

Nice. I don't remember manually setting that option. Is you mac Intel or M1/M2?

It is probably complaining at utils/astrometry.py:462. IOP4 creates a fork just to run the astrometry solver as a workaround for neuromorphicsystems/astrometry#6.

[morcuended]

M1

[juanep97]

I don't know how to avoid this error while forking a process in mac. Fork is better for this use case, since it is just a wrapper to run a function, and forking a process is much faster than spawning a new one. We could set this option inside IOP4 with setenv or just warn in the manual that it this environment variable might be needed?

[juanep97]

It might be solved with the right combinations of mac + cpython, since it seems related to the way cpython forks the process.

[morcuended]

I think a warning note in the installation instructions is sufficient.

[morcuended]

all tests passed!

[juanep97]

Instructions for tests added in PR #21.