Db export, again #88

danielballan · 2016-09-15T17:08:54Z

This is a rebase of #76 with some extra commits to add a test and make it pass.

codecov-io · 2016-09-15T17:21:28Z

Current coverage is 27.93% (diff: 2.63%)

Merging #88 into master will decrease coverage by 53.46%

@@             master        #88   diff @@
==========================================
  Files             6          6          
  Lines           661        691    +30   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits            538        193   -345   
- Misses          123        498   +375   
  Partials          0          0

Powered by Codecov. Last update d8022be...489fec5

danielballan · 2016-09-15T17:29:25Z

This may be ready. It could use a careful read from more than one reviewer. As you can see, the outline for copying external files is here, but not finished. (The docs include warning about that.) I'd like to roll this out as is and handle the external files piece in a second pass.

licode · 2016-09-15T19:35:40Z

databroker/broker.py

+                                      res['resource_path'],
+                                      res['resource_kwargs'],
+                                      root=new_root)
+                                      # FIXME: revisit root when dealing


The new_root is the root place where the external data file is going to transferred? So basically we can transfer data between different servers.

danielballan · 2016-09-15T19:49:02Z

@licode Yes, that is the plan. As written so far, this only copies database documents between servers. A future iteration will make use of the new_root argument to copy files as well.

licode · 2016-09-15T19:51:54Z

Got it. thanks.
Can we detect the root path automatically? So we do not need to manually change root, or keep this as an optional argument.

danielballan · 2016-09-15T20:49:30Z

How would that work?

Get Outlook for iOShttps://aka.ms/o0ukef

On Thu, Sep 15, 2016 at 3:52 PM -0400, "Li Li" <notifications@github.com mailto:notifications@github.com> wrote:

Got it. thanks.
Can we detect the root path automatically? So we do not need to manually change root, or keep this as an optional argument.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/NSLS-II/dataportal/pull/88#issuecomment-247433859, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACLIrh5G6yfFuReEvkfFjzXQYK6mgaOpks5qqaHbgaJpZM4J-G9z.

licode · 2016-09-15T20:58:26Z

I think I oversimplify the issue. Please ignore that.

licode · 2016-09-15T21:00:09Z

👍 This looks good to me

tacaswell · 2016-09-16T03:10:28Z

databroker/broker.py

+                db.fs.insert_resource(res['spec'],
+                                      res['resource_path'],
+                                      res['resource_kwargs'],
+                                      root=new_root)


This should pass through res['root'] until this is actually implemented.

Note to anyone following along: this is now actually implemented.

tacaswell · 2016-09-16T03:11:41Z

databroker/broker.py

+            headers = [headers]
+        for header in headers:
+            # insert mds
+            db.mds.insert_run_start(**_clean_doct(header['start']))


Why do you need this?

and by 'this' I mean _clean_doct

Because a doct has a _name attribute hanging off of it that gets unpacked in insert_run_start.

I am pretty sure we should get rid of '_name' -- an artifact of mongoengine documents -- but that's a question for another day.

tacaswell · 2016-09-16T03:12:20Z

databroker/broker.py

@@ -799,3 +855,10 @@ def _munge_time(t, timezone):
    """
    t = datetime.fromtimestamp(t)
    return timezone.localize(t).replace(microsecond=0).isoformat()
+
+
+def _clean_doct(d):


There is a function in doct for doing this splitting.

follow instruction on github and reference function signature

need to be reviewd and figure out exact input object

also update docstring and code acoordingly

danielballan · 2016-11-08T20:13:28Z

This is close, but it requires a FileStore that implements resource_given_eid. @tacaswell only implemented this method on FileStoreMoving, which means that users would need to set up FileStoreMoving on the production databroker before exporting. It seem this is actually encouraging the opposite of careful behavior, but I think @tacaswell has some opinions about these methods, so I'll leave the rest to him.

danielballan · 2016-11-10T14:57:50Z

If NSLS-II/filestore#121 is merged, this PR should just work. But I don't want to run roughshod over any concerns @tacaswell may have here, so please do not merge unless you are @tacaswell. :- )

danielballan · 2016-11-10T23:56:58Z

databroker/core.py

@@ -190,7 +191,7 @@ def get_events(mds, fs, headers, fields=None, stream_name=ALL, fill=False,
                # Look in the descriptor, then start, then stop.
                config_data_fields = set(filter(comp_re.match, config_data)) - selected_fields
                for field in config_data_fields:
-                    selected_fields.append(field)
+                    selected_fields.add(field)


This part is an unrelated bug fix.

danielballan · 2016-11-11T00:12:26Z

Note to whoever has the patience to finish this someday: FileStore.datum_gen_given_resource calls core.get_datumkw_by_resuid_gen which returns the kwargs from the datums but successfully obscures the datum_ids. FileStore is designed to make it hard to get this stuff, and that aspect of the design is proving to be a wild success.

danielballan · 2016-11-14T16:36:13Z

databroker/broker.py

+                                      res['resource_path'],
+                                      res['resource_kwargs'],
+                                      root=new_root)
+                datums = self.fs.datum_gen_given_resource(uid)


NSLS-II/filestore#122 will make this method,datum_gen_given_resource, return the full datum document (including the ID which we need) instead of just the datum kwargs.

danielballan · 2016-11-14T16:53:25Z

Test passes locally and should pass here once NSLS-II/filestore#122 and bluesky/bluesky#612 are in. Phew.

tacaswell · 2016-11-15T02:57:17Z

Not wanting to feel left out I added a commit to this PR too!

danielballan · 2016-11-15T04:00:35Z

Test failure is unrelated to this PR but probably real. I'll open a separate issue and expand on that.

Seeing @tacaswell's marked approval above, and knowing that three of us have code in here, I guess I'll be the one to pull the trigger....

CJ-Wright · 2016-11-15T16:00:08Z

databroker/broker.py

+        """
+        Export a list of headers.
+
+        .. warning::


Does this warning still apply?

It does not

danielballan force-pushed the db_export branch from 201e97f to 182b5cb Compare September 15, 2016 17:21

danielballan mentioned this pull request Sep 15, 2016

Db export #76

Closed

licode reviewed Sep 15, 2016

View reviewed changes

tacaswell reviewed Sep 16, 2016

View reviewed changes

tacaswell requested changes Sep 16, 2016

View reviewed changes

chiahaoliu and others added 14 commits November 8, 2016 15:10

init commit. signature and docstring

ca7c12e

First attempt of export function

7bf22ae

follow instruction on github and reference function signature

clean docstring of export function

a52bc19

middle of working on insert fs

ac821d9

finish preliminary method: just put on signature

c04c717

need to be reviewd and figure out exact input object

FIX: remove incorrect indentation and function signatures

7867928

FIX: update signature and typo in docstring

a0b5b2d

FIX: update signature of export method. It takes db now.

07719b5

also update docstring and code acoordingly

DOC: Fix docstring indentation and add warning about external files.

b0c263c

MNT: Accept one header or multiple headers.

2f28078

FIX: Fix typos and strip of doct-specific stuff before exporting.

15f9213

TST: Test db.export

2fe1ddc

MTN: remove _clean_doct and replace with method

924abe1

WIP: Add file-copying to db.export.

b769736

danielballan force-pushed the db_export branch from 050d8a3 to b769736 Compare November 8, 2016 20:11

danielballan mentioned this pull request Nov 10, 2016

MNT: Make resource_given_eid available on base FileStore. NSLS-II/filestore#121

Merged

danielballan closed this Nov 10, 2016

danielballan reopened this Nov 10, 2016

WIP

7b524a6

danielballan force-pushed the db_export branch from 173b54d to 7b524a6 Compare November 10, 2016 23:55

danielballan commented Nov 10, 2016

View reviewed changes

danielballan commented Nov 14, 2016

View reviewed changes

danielballan and others added 2 commits November 14, 2016 15:04

FIX: FileStore API does not give user control over resource id.

489fec5

TST: clean up mds databases

54bcac8

tacaswell approved these changes Nov 15, 2016

View reviewed changes

danielballan merged commit b2bb5fa into bluesky:master Nov 15, 2016

danielballan deleted the db_export branch November 15, 2016 04:00

danielballan mentioned this pull request Nov 15, 2016

Time-based search may be broken #102

Closed

CJ-Wright reviewed Nov 15, 2016

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Db export, again #88

Db export, again #88

danielballan commented Sep 15, 2016

codecov-io commented Sep 15, 2016 •

edited

Loading

danielballan commented Sep 15, 2016

licode Sep 15, 2016 •

edited

Loading

danielballan Nov 8, 2016

danielballan commented Sep 15, 2016

licode commented Sep 15, 2016

danielballan commented Sep 15, 2016

licode commented Sep 15, 2016

licode commented Sep 15, 2016

tacaswell Sep 16, 2016

danielballan Nov 14, 2016

tacaswell Sep 16, 2016

tacaswell Sep 16, 2016

danielballan Oct 7, 2016

danielballan Oct 7, 2016

tacaswell Sep 16, 2016

danielballan commented Nov 8, 2016

danielballan commented Nov 10, 2016

danielballan Nov 10, 2016

danielballan commented Nov 11, 2016

danielballan Nov 14, 2016

danielballan commented Nov 14, 2016

tacaswell commented Nov 15, 2016

danielballan commented Nov 15, 2016

CJ-Wright Nov 15, 2016

tacaswell Nov 15, 2016

Db export, again #88

Db export, again #88

Conversation

danielballan commented Sep 15, 2016

codecov-io commented Sep 15, 2016 • edited Loading

Current coverage is 27.93% (diff: 2.63%)

danielballan commented Sep 15, 2016

licode Sep 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielballan commented Sep 15, 2016

licode commented Sep 15, 2016

danielballan commented Sep 15, 2016

licode commented Sep 15, 2016

licode commented Sep 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielballan commented Nov 8, 2016

danielballan commented Nov 10, 2016

Choose a reason for hiding this comment

danielballan commented Nov 11, 2016

Choose a reason for hiding this comment

danielballan commented Nov 14, 2016

tacaswell commented Nov 15, 2016

danielballan commented Nov 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Sep 15, 2016 •

edited

Loading

licode Sep 15, 2016 •

edited

Loading