Always write to tar file in serial #197

spencerkclark · 2017-08-31T15:15:11Z

@spencerahill this is a possible really basic workaround for #75. I don't think writing to a tar file is a particular bottleneck in our pipeline (all desired results are already computed and saved out to files at that point), so one solution is to always write to the tar file in serial (even if the computations themselves are done in parallel).

What are your thoughts on this? I think we could revisit this problem if we have time in the future, but since it doesn't seem like there is a simple fail-proof way to enable writing to a tar file in parallel, it might be best just to avoid it.

spencerahill

I think we could revisit this problem if we have time in the future, but since it doesn't seem like there is a simple fail-proof way to enable writing to a tar file in parallel, it might be best just to avoid it.

I totally agree. Good thinking, and thanks for implementing it.

I gave a few minor comments, and can you add a what's new? In terms of tests, this seems like it would be difficult to test. I'm not convinced we need them here, but of course they'd be a nice addition if they are in fact easy enough.

spencerahill · 2017-08-31T17:48:44Z

aospy/automate.py

@@ -295,19 +295,28 @@ def _exec_calcs(calcs, parallelize=False, client=None, **compute_kwargs):
        def func(calc):
            """Wrap _compute_or_skip_on_error to require only the calc
            argument"""
-            return _compute_or_skip_on_error(calc, compute_kwargs)
+            return _compute_or_skip_on_error(calc, {'write_to_tar': False})


This causes none of the other compute_kwargs to be passed. Maybe update the value instead?

compute_kwargs.update({'write_to_tar': False}) _compute_or_skip_on_error(calc, compute_kwargs)

(I recognize that currently 'write_to_tar' is the only supported kwarg, so for now this is irrelevant. But this leaves the door open for other options in the future.)

spencerahill · 2017-08-31T17:56:51Z

aospy/automate.py

        else:
-            return _submit_calcs_on_client(calcs, client, func)
+            result = _submit_calcs_on_client(calcs, client, func)
+        _serial_write_to_tar(calcs, **compute_kwargs)


Similar to above comment: replace **compute_kwargs with write_to_tar=compute_kwargs['write_to_tar'].

spencerahill · 2017-08-31T17:59:59Z

aospy/automate.py

    else:
        return [_compute_or_skip_on_error(calc, compute_kwargs)
                for calc in calcs]

+def _serial_write_to_tar(calcs, write_to_tar=True):
+    if write_to_tar:


I think the logic is more intuitive if this if statement goes before the function call:

write_to_tar = compute_kwargs['write_to_tar'] if write_to_tar: _serial_write_to_tar(calcs, write_to_tar=write_to_tar)

Or, if you want to keep the if statement where it is, make the function name '_maybe_serial_write_to_tar'

(which supersedes the above comment on L307)

spencerkclark · 2017-08-31T19:23:57Z

Thanks @spencerahill, I agree with all your comments. I can't think of a great way to test this either (to some extent so long as our test suite doesn't produce empty header-related errors in the future, we should consider this PR to be successful).

spencerahill · 2017-08-31T20:04:25Z

Thanks @spencerkclark ! Now just need that ongoing dask.distributed bug for 2.7 to be fixed upstream, and we'll finally be back to getting passing test suites :)

chuaxr · 2017-11-08T23:26:13Z

While running the code for #228 I noticed a similar tar error:

Traceback (most recent call last):
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/site-packages/aospy/automate.py", line 253, in _compute_or_skip_on_error
    return calc.compute(**compute_kwargs)
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/site-packages/aospy/calc.py", line 626, in compute
    save_files=True, write_to_tar=write_to_tar)
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/site-packages/aospy/calc.py", line 709, in save
    self._write_to_tar(dtype_out_time)
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/site-packages/aospy/calc.py", line 663, in _write_to_tar
    with tarfile.open(self.path_tar_out, 'a') as tar:
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/tarfile.py", line 1606, in open
    return cls.taropen(name, mode, fileobj, **kwargs)
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/tarfile.py", line 1616, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/nbhome/xrc/anaconda2/envs/py361/lib/python3.6/tarfile.py", line 1493, in __init__
    raise ReadError(str(e))
tarfile.ReadError: empty header

It is possible that I am still seeing this message because I've been waiting for the custom reduction methods to be added before I upgrade. Does this mean that the data will occasionally fail to get written to tar, or can be it considered a false alarm?

spencerkclark · 2017-11-09T00:11:18Z

Yes, this seems like the same issue we resolved with this PR. Indeed, it means without this fix the data will occasionally fail to be written to the tar archive when calculations are submitted in parallel. Do you use the tar archive for anything? If you don't want to see these error messages, in lieu of updating to the master version of aospy and reimplementing your custom reduction method, one option would be to set write_to_tar in your main script to False. See here for example: https://github.com/spencerahill/aospy/blob/develop/aospy/examples/aospy_main.py#L123

chuaxr · 2017-11-09T00:25:12Z

I keep the tar files as a backup, so it would be nice to have write_to_tar as True. If the custom reductions are going to be added in the near future, I'd wait for that. Otherwise, I agree that one of the options you suggested would work.

spencerkclark · 2017-11-09T00:28:51Z

If the custom reductions are going to be added in the near future, I'd wait for that.

On my end probably not until January at the earliest :(

spencerahill · 2017-11-09T15:53:46Z

Unfortunately same here...I wouldn't count on those being implemented until 2018.

Always write to tar file in serial

36601c6

spencerahill reviewed Aug 31, 2017

View reviewed changes

This was referenced Aug 31, 2017

Retain original data's mask when yearly averaging #196

Merged

Towards v0.2 release #198

Closed

spencerkclark added 2 commits August 31, 2017 15:12

Merge branch 'develop' of git:spencerahill/aospy into fix-tar-issue

c226634

Address review comments

6bae024

spencerahill merged commit ce7c784 into spencerahill:develop Aug 31, 2017

spencerkclark deleted the fix-tar-issue branch August 31, 2017 20:07

spencerkclark mentioned this pull request Aug 31, 2017

Prevent Empty Header error when two processes try to write to same .tar file #75

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Always write to tar file in serial #197

Always write to tar file in serial #197

spencerkclark commented Aug 31, 2017

spencerahill left a comment

spencerahill Aug 31, 2017

spencerahill Aug 31, 2017

spencerahill Aug 31, 2017

spencerahill Aug 31, 2017

spencerkclark commented Aug 31, 2017

spencerahill commented Aug 31, 2017

chuaxr commented Nov 8, 2017

spencerkclark commented Nov 9, 2017

chuaxr commented Nov 9, 2017

spencerkclark commented Nov 9, 2017

spencerahill commented Nov 9, 2017

Always write to tar file in serial #197

Always write to tar file in serial #197

Conversation

spencerkclark commented Aug 31, 2017

spencerahill left a comment

Choose a reason for hiding this comment

spencerahill Aug 31, 2017

Choose a reason for hiding this comment

spencerahill Aug 31, 2017

Choose a reason for hiding this comment

spencerahill Aug 31, 2017

Choose a reason for hiding this comment

spencerahill Aug 31, 2017

Choose a reason for hiding this comment

spencerkclark commented Aug 31, 2017

spencerahill commented Aug 31, 2017

chuaxr commented Nov 8, 2017

spencerkclark commented Nov 9, 2017

chuaxr commented Nov 9, 2017

spencerkclark commented Nov 9, 2017

spencerahill commented Nov 9, 2017