ENH/API: Update database for moved files

[tacaswell]

Obviously a major work in progress

[ericdill]

filestore, not matplotlib

[ericdill]

The idea is that move_resource is a helper function that calls copy_ then remove_?

[ericdill]

Have you thought about how to implement the remove_? It seems like there are two legitimate options here.

1. Delete the Resource from the database immediately
2. Set a stale flag on the Resource to mark it for deletion at some later point. This would allow for an easy way to handle the 'oh shit I definitely did not mean to remove that resource!' at the expense of an increased complexity of the database

[tacaswell]

And with that I declare preliminary victory. Still to do:

• All of the handlers (that want to move their files) need to have a get_file_list method
• docs, all of the docs
• write migration script v0 -> v1 is back-compatible on read
• house keeping / error handling
• test coverage on the update log
• data broker tools to get all of the resources for a given runstart (maybe) ?
• by root and rpath searching of resources

@heroux proof of concept of moving files + updating the db to be aware of it.

[heroux]

my first @ mentioned ! Great!

[danielballan]

Consider updating this or moving it to a separate branch. Misinformation is worse than no information.

[tacaswell]

meh, it is pretty clearly labeled as a proposal and I think the outline of this is still correct long term, what is implemented is a fraction of this.

[danielballan]

fair enough

[danielballan]

@tacaswell Done with first round of review. Ping me when there are docstrings and I'll check those as well.

[tacaswell]

------------------------------------------------------------------------------------------------------- coverage: platform linux, python 3.5.1-final-0 -------------------------------------------------------------------------------------------------------
Name                              Stmts   Miss  Cover   Missing
---------------------------------------------------------------
filestore/api.py                     41      0   100%   
filestore/commands.py                 4      0   100%   
filestore/conf.py                    30      2    93%   47, 74
filestore/core.py                    96      2    98%   56, 164
filestore/core_v0.py                 70     11    84%   13-16, 40, 75, 88-91, 98
filestore/file_writers.py            53      0   100%   
filestore/fs.py                     267     38    86%   31-69, 93, 190, 193, 290, 375, 381, 386, 478, 481, 498
filestore/handlers.py               186     33    82%   26-30, 33-46, 49, 116, 178-181, 184-189, 254-256, 259-263, 303, 383, 400
filestore/handlers_base.py           13      0   100%   
filestore/path_only_handlers.py      22      1    95%   36
filestore/readers/__init__.py         0      0   100%   
filestore/readers/spe.py            162    126    22%   49-52, 56-100, 106, 109, 113, 117, 121-122, 126, 130, 134-139, 143, 147, 151, 159-162, 165-166, 169-170, 173, 176, 180-203, 206-213, 216-218, 221-235, 238-240, 245-247
filestore/retrieve.py                 7      0   100%   
filestore/utils.py                   24      8    67%   9-15, 42
---------------------------------------------------------------
TOTAL                               975    221    77%   

Most of these missing lines (excluding spe) are raised exceptions which are not exercised, or LPy work arounds

[danielballan]

?