Decouple job and mixins #624

larsbutler · 2011-12-29T11:55:15Z

This branch addresses: https://bugs.launchpad.net/openquake/+bug/908148

The major change here is that calculator Mixin classes must be instantiated with a Job object. The Job and the Mixin are no longer mashed together at runtime into a Frankenobject; instead, we use a silly little pattern called 'composition'.

There is still a bit of a mess here; the mixins still exist and some things are not quite how we would like them to be. This branch, however, is a big step towards our goal.

For a few of the next steps in Operation: Destroy the Mixins, see:
https://bugs.launchpad.net/openquake/+bug/909663
https://bugs.launchpad.net/openquake/+bug/907243

https://github.com/gem/openquake/pull/622 and https://github.com/gem/openquake/pull/623 should land first.

Now all of the calculators can start retrieving parameters self.job, instead of self. This is a good first step towards decouple the job and the calculator.

…rom the mixin/calculators Things are really messy right now I'll need at least 2 more branches: 1 for cleanup and 1 for implementing proper calculator structure

…nd-mixins

kpanic · 2012-01-04T16:41:54Z

pylint pick

Running pylint on larsbutler/decouple-job-and-mixins..

!! larsbutler/decouple-job-and-mixins adds 30 pylint issues..

``--- /tmp/pl.a 2012-01-04 17:39:02.000000000 +0100
+++ /tmp/pl.b 2012-01-04 17:39:15.000000000 +0100
@@ -2,0 +3 @@
+openquake/engine.py: [W0212, launch] Access to a protected member _record_initial_stats of a client class
@@ -9,0 +11,7 @@
+openquake/hazard/calc.py: [C0111] Missing docstring
+openquake/hazard/calc.py: [C0111, _load_calcs] Missing docstring
+openquake/hazard/opensha.py: [W0404] Reimport 'kvs' (imported line 38)
+openquake/hazard/opensha.py: [W0404] Reimport 'BasePSHAMixin' (imported line 45)
+openquake/hazard/opensha.py: [W0404] Reimport 'get_iml_list' (imported line 45)
+openquake/hazard/opensha.py: [W0404, compute_ground_motion_fields] Reimport 'CALCULATORS' (imported line 28)
+openquake/hazard/opensha.py: [W0404, compute_hazard_curve] Reimport 'CALCULATORS' (imported line 28)
@@ -11,0 +20,22 @@
+openquake/risk/calc.py: [C0111] Missing docstring
+openquake/risk/calc.py: [C0111, _load_calcs] Missing docstring
+openquake/risk/job/general.py: [W0404, compute_risk] Reimport 'CALCULATORS' (imported line 23)
+openquake/nrml/init.py: [R0801] Similar lines in 2 files
+==openquake.risk.job.classical_psha
+==openquake.risk.job.probabilistic

               for loss_poe in general.conditional_loss_poes(

                   self.job_profile.params):

                   general.compute_conditional_loss(

                           self.job_profile.job_id, point.column,

```
                           point.row, loss_curve, asset, loss_poe)
```
+openquake/nrml/init.py: [R0801] Similar lines in 2 files
+==openquake.risk.job.classical_psha
+==openquake.risk.job.probabilistic

   bcr_block_key = kvs.tokens.bcr_block_key(self.job_profile.job_id,

                                            block_id)

   kvs.set_value_json_encoded(bcr_block_key, result)

   LOGGER.debug('bcr result for block %s: %r', block_id, result)

```
   return True
```
+``

pep8 pick

Running pep8 on larsbutler/decouple-job-and-mixins..

openquake/output/template.py:23:80: E501 line too long (103 characters) openquake/output/template.py:89:80: E501 line too long (103 characters)

kpanic · 2012-01-04T16:56:06Z

Changes have to be done accordingly to celeryconfig.py

kpanic@hpinux:~/src/openquake-kpanic (bb)$ celeryd Traceback (most recent call last): File "/usr/local/bin/celeryd", line 9, in <module> load_entry_point('celery==2.3.1', 'console_scripts', 'celeryd')() File "/usr/lib/pymodules/python2.7/celery/bin/celeryd.py", line 187, in main worker.execute_from_commandline() File "/usr/lib/pymodules/python2.7/celery/bin/base.py", line 72, in execute_from_commandline return self.handle_argv(prog_name, argv[1:]) File "/usr/lib/pymodules/python2.7/celery/bin/base.py", line 100, in handle_argv return self.run(*args, **vars(options)) File "/usr/lib/pymodules/python2.7/celery/bin/celeryd.py", line 96, in run return self.app.Worker(**kwargs).run() File "/usr/lib/pymodules/python2.7/celery/apps/worker.py", line 124, in run self.worker_init() File "/usr/lib/pymodules/python2.7/celery/apps/worker.py", line 183, in worker_init self.loader.init_worker() File "/usr/lib/pymodules/python2.7/celery/loaders/base.py", line 84, in init_worker self.on_worker_init() File "/usr/lib/pymodules/python2.7/celery/loaders/default.py", line 54, in on_worker_init self.import_default_modules() File "/usr/lib/pymodules/python2.7/celery/loaders/base.py", line 79, in import_default_modules for module in imports | self.builtin_modules] File "/usr/lib/pymodules/python2.7/celery/loaders/base.py", line 67, in import_task_module return self.import_from_cwd(module) File "/usr/lib/pymodules/python2.7/celery/loaders/base.py", line 74, in import_from_cwd self.import_module if imp is None else imp) File "/usr/lib/pymodules/python2.7/celery/utils/__init__.py", line 407, in import_from_cwd return imp(module) File "/usr/lib/pymodules/python2.7/celery/loaders/base.py", line 70, in import_module return importlib.import_module(module) File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) ImportError: No module named tasks

larsbutler · 2012-01-05T10:36:28Z

I'm going to ignore the pep8 warnings in openquake/output/template.py; it's just a bunch of text for an html template (it's not code). Is that okay with you?

kpanic · 2012-01-05T10:38:50Z

yep

larsbutler · 2012-01-05T10:46:59Z

Cool.

Regarding the celeryd import issue, I'm not able to reproduce this error.. but I see the problem. Celery is importing stuff that isn't there anymore so I will update the config. I'll submit an update in just a second.

kpanic · 2012-01-05T10:52:51Z

Lars, maybe you have an old celeryconfig.pyc or tasks.pyc ?

try:

find -name \*pyc|xargs -i rm {}

This reverts commit aad949c.

al-maisan · 2012-01-10T11:50:13Z

More pylint fun:
openquake/logs.py: [E0202, AMQPHandler.emit] An attribute inherited from SupervisorLogMessageConsumer hide this method
openquake/output/init.py: [C0111] Missing docstring

al-maisan · 2012-01-10T11:50:42Z

pep8:

openquake/output/template.py:23:80: E501 line too long (103 characters)
openquake/output/template.py:89:80: E501 line too long (103 characters)

al-maisan · 2012-01-10T11:51:24Z

pyflakes:

tests/hazard_unittest.py:40: 'mixins' imported but unused
tests/hazard_unittest.py:47: 'openquake' imported but unused
tests/input_risk_unittest.py:24: 'Job' imported but unused
tests/input_risk_unittest.py:25: 'from openquake.output.hazard import *' used; unable to detect undefined names
tests/input_risk_unittest.py:30: 'Region' imported but unused
tests/job_unittest.py:38: 'Mixin' imported but unused
tests/job_unittest.py:41: 'general' imported but unused
tests/job_unittest.py:42: 'ProbabilisticEventMixin' imported but unused
tests/job_unittest.py:43: 'ClassicalPSHABasedMixin' imported but unused
tests/risk_job_unittest.py:25: 'job' imported but unused
tests/risk_job_unittest.py:29: 'Mixin' imported but unused
tests/utils/helpers.py:31: 'subprocess' imported but unused

al-maisan · 2012-01-10T11:52:14Z

Also:

openquake/job/__init__.py:23: 'urlparse' imported but unused
openquake/job/__init__.py:34: 'logs' imported but unused
openquake/job/__init__.py:43: 'Mixin' imported but unused

…nd-mixins

al-maisan · 2012-01-11T09:46:26Z

Typo in openquake/hazard/init.py

s/functionaity/functionality/

al-maisan · 2012-01-11T10:05:03Z

LGTM

[r=al-maisan] [f=908148] Decouple job and mixins

Implements the Shahjouei & Pezeshk (2016) GMM

larsbutler added 16 commits December 23, 2011 17:02

Added self.job member to Mixin base class

6ec0821

Now all of the calculators can start retrieving parameters self.job, instead of self. This is a good first step towards decouple the job and the calculator.

Ginormous refactoring job here. Job objects are now fully decoupled f…

b4ce1c8

…rom the mixin/calculators Things are really messy right now I'll need at least 2 more branches: 1 for cleanup and 1 for implementing proper calculator structure

Did some pep8 cleanup and deleted some unused modules

026b90b

pylint/pep8

f39006f

Merged master

e79a360

removed unused imports

29a40b0

pylint

5905554

pep8 cleanup

406200d

Removed some commented code.

a08c2c1

reverting some changes to db upgrade scripts

d6e0b3b

reverting some changes to db upgrade script

73cd204

reverting another change to db upgrade script

0bf35fc

Corrected a comment

b6ae330

Deleted an unused module (this code was moved to somewhere else)

710d08d

Fixed a circular import

82dccd6

Merge branch 'master' of github.com:gem/openquake into decouple-job-a…

d8d30db

…nd-mixins

larsbutler added 5 commits January 4, 2012 18:03

removed some superfluous calculator map code.. what was I thinking? =P

44938d2

pylint

13475fc

pylint

521339e

Merging master

796f276

pylint

afb7321

Fixed celeryconfig imports

5371707

larsbutler added 2 commits January 9, 2012 10:40

Reverting a change to upgrade script

aad949c

Revert "Reverting a change to upgrade script"

0863c78

This reverts commit aad949c.

larsbutler added 12 commits January 10, 2012 12:04

Deleted an unused module (this code was moved to somewhere else)

104bb73

Fixed a circular import

282110e

removed some superfluous calculator map code.. what was I thinking? =P

2ee6ee4

pylint

4d6c9b5

pylint

cd1e846

pylint

470ece4

Fixed celeryconfig imports

4d97fd6

Reverting a change to upgrade script

27fe910

Revert "Reverting a change to upgrade script"

2dacdb4

This reverts commit aad949c.

Fixed merge conflicts

064235c

pylint/pep8

ca52dfd

Merging master

5d8a0fa

larsbutler added 6 commits January 10, 2012 12:53

added missing docstring

e8450e9

silencing a false pylint warning

25b1bb2

pep8/pylint/pyflakes

d605853

Removed some superfluous code fron bin/openquake

0dfb2bb

Merge branch 'master' of github.com:gem/openquake into decouple-job-a…

b7054da

…nd-mixins

pyflakes

f2f0507

larsbutler added 3 commits January 11, 2012 12:17

Merged master

21a71f7

Fixing some bugs from merges

0d67444

fixed a typo

c5b714b

larsbutler added a commit that referenced this pull request Jan 11, 2012

Merge pull request #624 from larsbutler/decouple-job-and-mixins

a77266b

[r=al-maisan] [f=908148] Decouple job and mixins

larsbutler merged commit a77266b into gem:master Jan 11, 2012

daniviga pushed a commit that referenced this pull request Jun 7, 2017

Merge pull request #624 from gem/shahjouei-pezeshk-2016

048032e

Implements the Shahjouei & Pezeshk (2016) GMM

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decouple job and mixins #624

Decouple job and mixins #624

larsbutler commented Dec 29, 2011

kpanic commented Jan 4, 2012

kpanic commented Jan 4, 2012

larsbutler commented Jan 5, 2012

kpanic commented Jan 5, 2012

larsbutler commented Jan 5, 2012

kpanic commented Jan 5, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 11, 2012

al-maisan commented Jan 11, 2012

Decouple job and mixins #624

Decouple job and mixins #624

Conversation

larsbutler commented Dec 29, 2011

kpanic commented Jan 4, 2012

kpanic commented Jan 4, 2012

larsbutler commented Jan 5, 2012

kpanic commented Jan 5, 2012

larsbutler commented Jan 5, 2012

kpanic commented Jan 5, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 10, 2012

al-maisan commented Jan 11, 2012

al-maisan commented Jan 11, 2012