Dump and restore Stochastic Event Set

[matley]

Addresses https://bugs.launchpad.net/oq-engine/+bug/1247665

Additionally, I have changed the database permissions such that oq_admin is able to create temporary tables (used for the dump/restore workflow).

For a clean diff, #1298 should land first

[micheles]

This branch should be refreshed

[matley]

Just refreshed

[micheles]

The first thing that I do not like is that now a regular user gets the error:

psycopg2.ProgrammingError: must be superuser to COPY to or from a file
HINT: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.

It should be possible to avoid that by using psycopg2 copy_expert functionality

[micheles]

What's the reason why we are not generating .gz files? This makes a lot of difference in terms of disk space occupation. Notice that this tool is intended to be used even for large outputs.

[matley]

This work is experimental. So, I did not want to add too many feature. I prefer at this moment to let the user zip or not zip the result. however, I see where you are coming from. Therefore, I have added a ticket https://bugs.launchpad.net/oq-engine/+bug/1248584

[matley]

I have just addressed all the raised issues, including the permission ones (by using copy_expert as you suggested). Could you give it a look again?

[micheles]

There is still something not working:

$ bin/openquake --restore-hazard-calculation /tmp
Traceback (most recent call last):
File "bin/openquake", line 546, in
main()
File "bin/openquake", line 539, in main
connection.connection, args.restore_hazard_calculation)
File "/home/michele/oq-engine/openquake/engine/tools/restore_hazards.py", line 147, in hazard_restore
curs = conn.cursor()
AttributeError: 'NoneType' object has no attribute 'cursor'

[micheles]

I get this error:

Traceback (most recent call last):
File "/home/michele/oq-engine/bin/openquake", line 545, in
main()
File "/home/michele/oq-engine/bin/openquake", line 535, in main
dump_hazards.main(*args.dump_hazard_calculation)
File "/home/michele/oq-engine/openquake/engine/tools/dump_hazards.py", line 274, in main
host=host, database=dbname, user=user, password=password, port=port)
File "/usr/lib/python2.7/dist-packages/psycopg2/init.py", line 167, in connect
if port is not None and int(port) > 0:
ValueError: invalid literal for int() with base 10: ''

The fix is to change engine/settings.py and set the port to '5432' instead of ''. How comes it works on your machine?

[micheles]

Then I get another error:
$ oq --restore-hazard-calculation /tmp/
Traceback (most recent call last):
File "/home/michele/oq-engine/bin/openquake", line 545, in
main()
File "/home/michele/oq-engine/bin/openquake", line 538, in main
models.getcursor('admin').connection, args.restore_hazard_calculation)
File "/home/michele/oq-engine/openquake/engine/tools/restore_hazards.py", line 158, in hazard_restore
hc_ids = transfer_data(curs, models.HazardCalculation)
File "/home/michele/oq-engine/openquake/engine/tools/restore_hazards.py", line 63, in transfer_data
"ALTER TABLE %s ADD restore_id INT" % model_table(model))
psycopg2.ProgrammingError: must be owner of relation hazard_calculation

[micheles]

After yesterday discussion, I think we should change the names to --save-hazard-calculation and --load-hazard-calculation. Paul and Matteo like those names better. Also restore_hazards.py and dump_hazards.py could be renamed.

[matley]

I have renamed dump/restore -> save/load. I have added a proper migration script to fix permission problems.

[micheles]

Notice that there is still the issue with the port number in engine/settings.py. Moreover I cannot dump Miriam's computation:

$ bin/openquake --shc 1 /tmp/
Dumping ses (1) in /tmp/
Dumping gmf (2) in /tmp/
Traceback (most recent call last):
File "/home/michele/oq-engine/bin/openquake", line 545, in
main()
File "/home/michele/oq-engine/bin/openquake", line 535, in main
save_hazards.main(*args.save_hazard_calculation)
File "/home/michele/oq-engine/openquake/engine/tools/save_hazards.py", line 276, in main
hc.dump(hazard_calculation_id)
File "/home/michele/oq-engine/openquake/engine/tools/save_hazards.py", line 246, in dump
self.gmf(ids)
File "/home/michele/oq-engine/openquake/engine/tools/save_hazards.py", line 175, in gmf
'hzrdr.gmf.csv', mode='a')
File "/home/michele/oq-engine/openquake/engine/tools/save_hazards.py", line 121, in _copy
self.outdir, filename, mode)
File "/home/michele/oq-engine/openquake/engine/tools/save_hazards.py", line 88, in copy
self._cursor.copy_expert(query, fileobj)
psycopg2.ProgrammingError: syntax error at or near ")"
LINE 1: copy (select * from hzrdr.gmf where output_id in (2)))

You should also refresh the branch from master.

[matley]

I have just fixed the issue with inputs having ground motion fields. I can't reproduce the problem you are experiencing (the one related to the settings.py). GMFs are currently unsupported, so now we skip processing them and warn the user properly.

[matley]

Hey Michele, look at the latest commit. I have also added support for gzip.

[micheles]

I have an issue with the ALTER TABLE locking the original tables, but it is not a problem when loading a calculation on an user computer (there is a single user there) and the approach can be revised later on, when this feature will be used in the platform. So LGTM for now.