[IO-1279] Functions to move items around stages #640

Nathanjp91 · 2023-07-21T14:02:03Z

Problem

Don't have any way to move items around workflows in darwin-py at the moment

Solution

Add in functions to stage items as well as backend core functions to move items

Changelog

Can move items in workflows, see item's attached to a stage and manage workflows better

linear · 2023-07-21T14:02:05Z

IO-1279 Core Backend: Push Item to Stage

Need to scope out the API required

Nathanjp91 · 2023-07-21T14:04:06Z

darwin/future/core/client.py

@@ -155,7 +155,7 @@ def _setup_session(self, retries: Retry) -> None:

    @property
    def headers(self) -> Dict[str, str]:
-        http_headers: Dict[str, str] = {"Content-Type": "application/json"}
+        http_headers: Dict[str, str] = {"Content-Type": "application/json", "Accept": "application/json"}


This was required for the stage api, but if we need to set this for other api's we'll need to think about a syntax for setting headers of the client, either passing headers as an optional parameter to the client.get(...)/REST functions or changing headers before certain API calls

Nathanjp91 · 2023-07-21T14:04:26Z

darwin/future/core/client.py

@@ -172,7 +172,7 @@ def _generic_call(self, method: Callable, endpoint: str, payload: Optional[dict]
        endpoint = self._sanitize_endpoint(endpoint)
        url = self.config.api_endpoint + endpoint
        if payload is not None:
-            response = method(url, payload)
+            response = method(url, json=payload)


data didn't work here for some reason

I think it only works if the data type isn't "app/json" or on certain methods.

Nathanjp91 · 2023-07-21T14:05:34Z

darwin/future/core/types/query.py

@@ -193,5 +193,21 @@ def where(self, *args: object, **kwargs: str) -> Query[T]:
    def collect(self) -> List[T]:
        raise NotImplementedError("Not implemented")

+    def collect_one(self) -> T:


Both here added as dogfooding session with Simon

Nathanjp91 · 2023-07-21T14:07:06Z

darwin/future/data_objects/workflow.py

@@ -173,7 +179,7 @@ class Workflow(DefaultDarwin):
    inserted_at: datetime
    updated_at: datetime

-    dataset: WFDataset
+    dataset: Optional[WFDataset]


Some workflows on platform don't have a dataset attached, prevents collecting these items if it's required field. Although intuitively you would think it should be

Yep, intuitively I did think so! Good catch

Nathanjp91 · 2023-07-21T14:12:12Z

darwin/future/meta/objects/base.py



 class MetaBase(Generic[R]):
    _item: Optional[R]
    client: Client

-    def __init__(self, client: Client, item: Optional[R] = None) -> None:
+    def __init__(self, client: Client, item: Optional[R] = None, meta_params: Optional[Param] = None) -> None:


meta_params have to be added to the MetaBase here so that chaining can happen properly with backend API calls.
client.team.workflows.first().stages.first().push_items_to_next_stage()

pushing items to stage requires other parameters like workflow_id, dataset_id, team_slug. This need to be passed from Meta -> Query -> Meta as it goes down the chain

owencjones

Yeah, looks good - I've commented a lot, but there's nothing that needs changing.

owencjones · 2023-07-25T09:34:52Z

darwin/future/core/client.py

@@ -172,7 +172,7 @@ def _generic_call(self, method: Callable, endpoint: str, payload: Optional[dict]
        endpoint = self._sanitize_endpoint(endpoint)
        url = self.config.api_endpoint + endpoint
        if payload is not None:
-            response = method(url, payload)
+            response = method(url, json=payload)


I think it only works if the data type isn't "app/json" or on certain methods.

owencjones · 2023-07-25T09:36:39Z

darwin/future/core/items/get.py

+        f"/v2/teams/{team_slug}/items/ids",
+        QueryString({"workflow_stage_ids": str(stage_id), "dataset_ids": str(dataset_id)}),
+    )
+    assert type(response) == dict


Could consider using the other function we have for this which is impervious to __DEBUG__ flag, but not the most important now - esp if this is just for making mypy happy!

owencjones · 2023-07-25T09:37:40Z

darwin/future/core/items/get.py

+        f"/v2/teams/{team_slug}/items/ids",
+        QueryString({"not_statuses": "archived,error", "sort[id]": "desc", "dataset_ids": str(dataset_id)}),
+    )
+    assert type(response) == dict


Same as my other (confusingly later) comment - could be assertis or just an if, but dw if this is just mypy cheering.

it's mostly for mypy type checking, but we can use the assert_is in these cases too.

owencjones · 2023-07-25T09:38:29Z

darwin/future/core/items/move_items.py

+
+    Returns
+    -------
+    None


Doesn't return None

owencjones · 2023-07-25T09:40:26Z

darwin/future/data_objects/workflow.py

@@ -173,7 +179,7 @@ class Workflow(DefaultDarwin):
    inserted_at: datetime
    updated_at: datetime

-    dataset: WFDataset
+    dataset: Optional[WFDataset]


Yep, intuitively I did think so! Good catch

owencjones · 2023-07-25T09:42:22Z

darwin/future/meta/objects/base.py

        self.client = client
        self._item = item
+        self.meta_params = meta_params or {}


There's some argument for this being or dict() just because {} is also the syntax for set delimitation, but now I'm really scraping the barrel I guess.

owencjones · 2023-07-25T09:44:06Z

darwin/future/meta/objects/dataset.py

@@ -1,10 +1,17 @@
-from typing import List, Optional, Tuple, Union
+from __future__ import annotations


With this, I tend to just quote the returnable I need it for, but this is valid too, especially with our current support matrix.

owencjones · 2023-07-25T09:45:43Z

darwin/future/meta/objects/dataset.py

@@ -155,3 +178,20 @@ def _validate_slug(slug: str) -> None:

        VALID_SLUG_CHARS = "abcdefghijklmnopqrstuvwxyz0123456789-_"
        assert_is(all(c in VALID_SLUG_CHARS for c in slug_copy), "slug must only contain valid characters")
+
+    def upload_files(


Good move using the CLI function - I was looking at which to do when I went sick last week.

owencjones · 2023-07-25T09:49:19Z

darwin/future/meta/objects/workflow.py

+
+    def push_from_dataset_stage(self) -> WorkflowMeta:
+        assert self._item is not None


Are all of these to satiate mypy?

pretty much

owencjones · 2023-07-25T09:50:55Z

darwin/future/meta/queries/stage.py

@@ -14,9 +14,10 @@ def collect(self) -> List[StageMeta]:
        if not self.meta_params:
            raise ValueError("Must specify workflow_id to query stages")
        workflow_id: UUID = self.meta_params["workflow_id"]
+        meta_params = self.meta_params.copy()


Pass-by-ref was mutilating params as they were passed around?

Nathanjp91 and others added 7 commits July 19, 2023 22:24

WIP: moving items to stage

6608ade

WIP: moving items to stage

f889e0c

WIP: moving items to stage

e8fb31d

pushing items between stages

d87f607

added stage syntax

2774558

WIP dogfooding functions

dc68710

fixing broken tests

60ac44d

Nathanjp91 commented Jul 21, 2023

View reviewed changes

Nathanjp91 added 4 commits July 24, 2023 09:52

remove curses

6700ed8

tests for query

ad487d5

tests for core

1612f2f

tests for stageMeta

05bbb90

Nathanjp91 marked this pull request as ready for review July 25, 2023 09:23

owencjones approved these changes Jul 25, 2023

View reviewed changes

Nathanjp91 added 3 commits July 25, 2023 20:51

dogfooding feedback

73c6153

dogfooding feedback

fcdc02e

fixes for tests

9b4a1b3

Nathanjp91 merged commit fae6e41 into master Jul 25, 2023
9 checks passed

Nathanjp91 deleted the IO-1279 branch November 8, 2023 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IO-1279] Functions to move items around stages #640

[IO-1279] Functions to move items around stages #640

Nathanjp91 commented Jul 21, 2023

linear bot commented Jul 21, 2023

Nathanjp91 Jul 21, 2023

Nathanjp91 Jul 21, 2023

owencjones Jul 25, 2023

Nathanjp91 Jul 21, 2023

Nathanjp91 Jul 21, 2023

owencjones Jul 25, 2023

Nathanjp91 Jul 21, 2023

owencjones left a comment

owencjones Jul 25, 2023

owencjones Jul 25, 2023

owencjones Jul 25, 2023

Nathanjp91 Jul 25, 2023

owencjones Jul 25, 2023

owencjones Jul 25, 2023

owencjones Jul 25, 2023

owencjones Jul 25, 2023

owencjones Jul 25, 2023

owencjones Jul 25, 2023

Nathanjp91 Jul 25, 2023

owencjones Jul 25, 2023

		@@ -1,10 +1,17 @@
		from typing import List, Optional, Tuple, Union
		from __future__ import annotations


		def push_from_dataset_stage(self) -> WorkflowMeta:
		assert self._item is not None

[IO-1279] Functions to move items around stages #640

[IO-1279] Functions to move items around stages #640

Conversation

Nathanjp91 commented Jul 21, 2023

Problem

Solution

Changelog

linear bot commented Jul 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owencjones left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment