Save ZTF alerts as sources by objectId #44

dmitryduev · 2020-09-29T08:29:45Z

In this PR:

extensions/skyportal/skyportal/handlers/api/alert.py:
- new endpoint ZTFAlertHandler.post to save ZTF alerts as sources by their objectId, e.g. api('POST', '<fritz_instance_location>/api/alerts/ztf/<objectId>'), {"candid": <candid>, "group_ids": <list of group ids>}. Under the hood, F pulls the required data from K, massages them, and posts to SP's /api/sources, /api/photometry, and /api/thumbnail. If candid is not provided, metadata and thumbnails will be pulled from the latest alert/candid. If no list of group_ids is provided, the object will be saved to all user's groups
- async api calls to Kowalski using tornado.httpclient instead of requests
extensions/skyportal/static/js/components/ZTFAlert.jsx
- Re-skin that makes the page look a lot more like the Source and Candidate pages
- Replaced MUI table with MUI datatable to display the alerts
- A button to save alerts as sources
extensions/skyportal/static/js/components/SaveAlertButton.jsx
- Mostly a copy-paste of SaveCandidateButton.jsx with a couple modifications necessary here
fix linting in extensions/skyportal/static/js/components/Filter.jsx

dmitryduev · 2020-09-29T21:55:39Z

extensions/skyportal/skyportal/handlers/api/alert.py

+            w = df["candid"] == str(candid)
+
+            if candid is None or sum(w) == 0:
+                candids = {int(can) for can in df["candid"] if not pd.isnull(can)}


Got sick of pandas relationship with long ints and implemented it this way.

dmitryduev · 2020-09-29T21:57:05Z

extensions/skyportal/skyportal/handlers/api/alert.py

+                log(f"Failed to post photometry of {objectId}")
+            # do not return anything yet
+            self.clear()
+


Monsieur knows a lot about perversion, don't you think @dannygoldstein? :) Hope you appreciate the "beauty"!

dmitryduev · 2020-09-29T21:57:49Z

extensions/skyportal/skyportal/handlers/api/alert.py

+            except Exception:
+                log(f"Failed to post photometry of {objectId}")
+            # do not return anything yet
+            self.clear()


If you don't do this, the photometry handler would try to .write()

dannygoldstein · 2020-09-30T03:13:28Z

merge conflicts

dannygoldstein

Main comment is: what happens when a user attempts to save an alert to a group without permissions to see the data or that the object exists?

E.g., an object with all PID=3 alerts saved to the sitewide_group

dannygoldstein · 2020-09-30T03:37:32Z

extensions/skyportal/static/js/components/SaveAlertButton.jsx

+import * as sourcetActions from "../ducks/source";
+import FormValidationError from "./FormValidationError";
+
+const SaveAlertButton = ({ alert, userGroups }) => {


What happens when a user tries to save an object with only PID=3 alerts to the sitewide group?

Great point, addressed here. It will now fail.

dannygoldstein · 2020-09-30T03:44:17Z

extensions/skyportal/skyportal/handlers/api/alert.py

@@ -141,10 +207,313 @@ def get(self, objectId: str = None):
            _err = traceback.format_exc()
            return self.error(f'failure: {_err}')

+    @auth_or_token
+    async def post(self, objectId):


because I am calling K asynchronously and awaiting the results; you can await only inside an async function.

dannygoldstein · 2020-09-30T03:45:35Z

extensions/skyportal/skyportal/handlers/api/alert.py

+                if len(alert_data) > 0:
+                    alert_data = alert_data[0]
+                else:
+                    return self.error(f"{objectId} not found on Kowalski")


As above, what about the case where someone saves an object to a group without permissions to see the data?

dannygoldstein · 2020-09-30T03:46:15Z

extensions/skyportal/skyportal/handlers/api/alert.py

+            ztf_filters = {1: 'ztfg', 2: 'ztfr', 3: 'ztfi'}
+            df['ztf_filter'] = df['fid'].apply(lambda x: ztf_filters[x])
+            df['magsys'] = "ab"
+            df['zp'] = 25.0


is this used downstream?

No, removed.

dmitryduev · 2020-09-30T07:06:20Z

@dannygoldstein Feedback addressed, please re-review.

kmshin1397

Some minor comments but otherwise looks good

kmshin1397 · 2020-09-30T17:49:55Z

extensions/skyportal/skyportal/handlers/api/alert.py

+                group_ids = [
+                    int(_id)
+                    for _id in group_ids
+                    if int(_id) in user_accessible_group_ids


If I understand this correctly, it seems like we just silently ignore any groups the user doesn't have access to. Should we send a specific error if a group_id passed in isn't part of the user_accessilbe_group_ids? To let someone know if they think they're part of a group but aren't for whatever reason?

Added a check on that

kmshin1397 · 2020-09-30T18:09:26Z

extensions/skyportal/static/js/components/SaveAlertButton.jsx

+import { useForm, Controller } from "react-hook-form";
+
+import * as alertActions from "../ducks/alert";
+import * as sourcetActions from "../ducks/source";


Suggested change

import * as sourcetActions from "../ducks/source";

import * as sourceActions from "../ducks/source";

kmshin1397 · 2020-09-30T18:09:49Z

extensions/skyportal/static/js/components/SaveAlertButton.jsx

+    } else {
+      setDialogOpen(false);
+      reset();
+      await dispatch(sourcetActions.fetchSource(alert.id));


Suggested change

await dispatch(sourcetActions.fetchSource(alert.id));

await dispatch(sourceActions.fetchSource(alert.id));

dannygoldstein

LGTM

dmitryduev added 7 commits September 27, 2020 03:18

use mui-datatable in ZTFAlert.jsx

f9272f2

make ZTFAlert look more Source

bb438fa

WIP: save alert as source

966e331

backend for saving alerts as sources

06f5f86

backend for saving alerts as sources; async call K

a7468f8

fix linting

74a1e0a

final touches

e3c6bcc

dmitryduev changed the title ~~Save ZTF alerts by objectId~~ Save ZTF alerts as sources by objectId Sep 29, 2020

dmitryduev added 5 commits September 29, 2020 13:58

hide save button upon success

8ffa4cb

fix linting

8ce2380

fix linting

54c6015

fix linting

20ff375

return error if objectId not found on K

b6bde4a

dmitryduev commented Sep 29, 2020

View reviewed changes

fix async-related issues

402b96f

dmitryduev requested review from dannygoldstein and acrellin September 29, 2020 23:21

dmitryduev marked this pull request as ready for review September 29, 2020 23:21

dmitryduev mentioned this pull request Sep 29, 2020

Add ability to save alerts as candidates with one click/api call #45

Closed

dmitryduev added 2 commits September 29, 2020 17:29

fix fritz-marshal#46

8eaa1a6

fix fritz-marshal#46

4d0cf5d

dmitryduev linked an issue Sep 30, 2020 that may be closed by this pull request

Command "./fritz run --init" broken due to recent config changes #46

Closed

dannygoldstein requested changes Sep 30, 2020

View reviewed changes

fix merge conflict due to fritz-marshal#42

eba0ff0

dmitryduev requested a review from kmshin1397 September 30, 2020 05:00

dmitryduev added 4 commits September 29, 2020 22:03

fix merge conflict due to fritz-marshal#42

8b73532

update SP

db3988b

update K

8ee0bc8

check group stream access permissions

947934a

dmitryduev added 2 commits September 29, 2020 23:59

remove option to save to all user groups by default

f599ebe

remove zp

d5827ad

kmshin1397 approved these changes Sep 30, 2020

View reviewed changes

dannygoldstein approved these changes Sep 30, 2020

View reviewed changes

dmitryduev and others added 2 commits September 30, 2020 12:33

Merge branch 'master' into save_alert_by_id

8c401b1

throw error if trying to post to groups unauthorized

1d3fa64

dmitryduev requested review from kmshin1397 and removed request for acrellin September 30, 2020 19:55

kmshin1397 approved these changes Sep 30, 2020

View reviewed changes

kmshin1397 merged commit bd87afe into fritz-marshal:master Sep 30, 2020

dmitryduev linked an issue Oct 6, 2020 that may be closed by this pull request

save source by source ID #39

Closed

dmitryduev mentioned this pull request Oct 6, 2020

save source by source ID #39

Closed

dmitryduev deleted the save_alert_by_id branch February 24, 2021 23:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Save ZTF alerts as sources by objectId #44

Save ZTF alerts as sources by objectId #44

dmitryduev commented Sep 29, 2020 •

edited

Loading

dmitryduev Sep 29, 2020

dmitryduev Sep 29, 2020

dmitryduev Sep 29, 2020

dannygoldstein commented Sep 30, 2020

dannygoldstein left a comment •

edited

Loading

dannygoldstein Sep 30, 2020

dmitryduev Sep 30, 2020

dannygoldstein Sep 30, 2020

dmitryduev Sep 30, 2020

dannygoldstein Sep 30, 2020

dannygoldstein Sep 30, 2020

dmitryduev Sep 30, 2020

dmitryduev commented Sep 30, 2020

kmshin1397 left a comment

kmshin1397 Sep 30, 2020

dmitryduev Sep 30, 2020

kmshin1397 Sep 30, 2020

dmitryduev Sep 30, 2020

kmshin1397 Sep 30, 2020

dmitryduev Sep 30, 2020

dannygoldstein left a comment

	import * as sourcetActions from "../ducks/source";
	import * as sourceActions from "../ducks/source";

	await dispatch(sourcetActions.fetchSource(alert.id));
	await dispatch(sourceActions.fetchSource(alert.id));

Save ZTF alerts as sources by objectId #44

Save ZTF alerts as sources by objectId #44

Conversation

dmitryduev commented Sep 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannygoldstein commented Sep 30, 2020

dannygoldstein left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitryduev commented Sep 30, 2020

kmshin1397 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannygoldstein left a comment

Choose a reason for hiding this comment

dmitryduev commented Sep 29, 2020 •

edited

Loading

dannygoldstein left a comment •

edited

Loading