Automatic jobs & new stats

[antgonza]

This adds automatic jobs (which should be added to a cron job by the sys admin) and new stats.

[coveralls]

Coverage decreased (-0.02%) to 91.509% when pulling 11f3657 on antgonza:fix-814 into 131dd6a on biocore:master.

[antgonza]

Tested in the test-env and seems to be working fine, runs < 2min. Once, we agree on what other stats we want to add (via the internal email) I will add the code so stats run from the new redis variables so nothing is calculated on the fly.

Also, this is how the storage plot looks like:

[coveralls]

Coverage decreased (-0.02%) to 91.51% when pulling 148cef5 on antgonza:fix-814 into 131dd6a on biocore:master.

[coveralls]

Coverage decreased (-0.07%) to 91.459% when pulling 9e9eee7 on antgonza:fix-814 into 131dd6a on biocore:master.

[coveralls]

Coverage decreased (-0.07%) to 91.459% when pulling 3ccce3d on antgonza:fix-814 into 131dd6a on biocore:master.

[antgonza]

This is ready for review, full code deployed in the test env.

[josenavas]

Overall looks good, few comments. Thanks @antgonza !

[josenavas]

I would suggest using "SampleTemplate.ebi_sample_accessions" to count the number of samples that are in EBI. You can double-count some of the samples here since they can have been sequenced more than one time.

If you still want to show this count, you can show "sample runs submitted to ebi". In that case you will have the number of samples, the number of samples in EBI and the number of runs in EBI.

[antgonza]

Excellent point, will add both.

[josenavas]

Note that in this for loop, each key enters in a single value of the if, which means that a for loop is not needed. I think a better way of doing this is:

keys = ['number_studies', 'number_of_samples', 'num_users', 'lat_longs', 'num_studies_ebi', 'num_samples_ebi', 'img', 'time']
values = [number_studies, number_of_samples, num_users, lat_longs, num_studies_ebi, num_samples_ebi, img, time]
funcs = [r_client.hmset, r_client.hmset, r_client.set, r_client.set, r_client.set, r_client.set, r_client.set, r_client.set]

for k, v, f in zip(keys, values, funcs):
    redis_key = '%s:stats:%s' % (portal, k)
    f(redis_key, v)

[josenavas]

Same comment as above re doing the for loop.

[josenavas]

Can you update the docstring with the new parameter?

[josenavas]

I don't think you need to pass TRN as a parameter, given that it is a global variable in the system, you can just use it.

[antgonza]

Turns out that you do, if not:

======================================================================
ERROR: test_purge_filepaths_null_cols (qiita_core.util.DecoratedClass)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "qiita_db/test/test_util.py", line 393, in test_purge_filepaths_null_cols
    self._common_purge_filpeaths_test()
  File "qiita_db/test/test_util.py", line 342, in _common_purge_filpeaths_test
    qdb.util.purge_filepaths()
  File "/Users/antoniog/svn_programs/qiita/qiita_db/util.py", line 768, in purge_filepaths
    _rm_files(fp)
  File "/Users/antoniog/svn_programs/qiita/qiita_db/util.py", line 724, in _rm_files
    TRN.add_post_commit_func(func, fp)
NameError: global name 'TRN' is not defined

[josenavas]

Oh, well, yeah, in that case you need to use it as:
qdb.sql_connection.TRN not just TRN.

[josenavas]

Not blocking though...

[ElDeveloper]

From looking at the file, I think this is because you need to import the TRN object, right @josenavas?

[antgonza]

Gonna leave as is ... as @josenavas said, not blocking.

[ElDeveloper]

Sounds good.

[josenavas]

Can you put each of this values in one row? I think it looks better than one next to each other.

[josenavas]

Why do you have this intermediate functions? You can call them directly under main.

[coveralls]

Coverage decreased (-0.08%) to 91.45% when pulling ace1d7a on antgonza:fix-814 into 131dd6a on biocore:master.

[coveralls]

Coverage decreased (-0.01%) to 91.519% when pulling b6cd23f on antgonza:fix-814 into 131dd6a on biocore:master.

[ElDeveloper]

Looks good overall!

[ElDeveloper]

This info seems handy, maybe would be worth to describe this in the help text for this script, so maybe just make this a small click command that can be executed without arguments? Not blocking though.

[antgonza]

thought about it but went with the simplest possible solution, which can grow as we see fit; rather leave it as this.

[ElDeveloper]

ok

[ElDeveloper]

I wonder if these numbers are big enough that they should be comma-separated by the thousands, for example like this:

>>> print "{:,}".format(11111111)
11,111,111

[antgonza]

good idea.

[ElDeveloper]

Would it make more sense to use SVG, that way the browser can scale it as appropriate and it should look as sharp as possible? 📈

[antgonza]

that could work but the issue is that then you will have to make a lot of changes to make the size of the image to fix in the page, IMOO not worth it.

[ElDeveloper]

Sounds good then.

[ElDeveloper]

Can you specify the exception name here?

[antgonza]

k

[ElDeveloper]

From looking at the file, I think this is because you need to import the TRN object, right @josenavas?

[ElDeveloper]

Is this print statement needed? Doesn't seem to be described in the docs.

[antgonza]

just added it.

[ElDeveloper]

Same as the other print.

[coveralls]

Coverage decreased (-0.02%) to 91.515% when pulling 640de3a on antgonza:fix-814 into 131dd6a on biocore:master.

[ElDeveloper]

Sounds good then.