[PLT-X] Make export_v2 methods use streamable backend #1515
Conversation
| filters: Union[CatalogExportFilters, Dict[str, List[str]], None] = None, | ||
| params: Optional[CatalogExportParams] = None, | ||
| ) -> Task: | ||
| ) -> Union[Task, ExportTask]: |
There was a problem hiding this comment.
In this PR we return a union or a tuple in several places, is it possible to avoid that ? Here for instance I can't tell the difference between Task and ExportTask. Also we end up not using one of the two return values in other places.
There was a problem hiding this comment.
This PR modifies ExportTask so that it is a superset of Task and behaves similarly. We cannot know what is being returned until we toggle the flag for all customers. We'll be able to narrow down the type in future version (once flag is enabled for everyone).
mnoszczak
force-pushed
the
mno/exports-v2-streamable
branch
from
f942843 to
6096995
Compare
mnoszczak
force-pushed
the
mno/exports-v2-streamable
branch
from
aeeb8d0 to
02aa1ed
Compare
|
|
||
| def __init__(self, task: Task) -> None: | ||
| def __init__(self, task: Task, is_export_v2: bool = False) -> None: | ||
| self._is_export_v2 = is_export_v2 |
There was a problem hiding this comment.
I assume we need to explicitly pass it in because of the api feature flag that might change streamable to export_v2?
There was a problem hiding this comment.
This value comes down from API though. The flow is as follows:
- New SDK version: Hey API I'm able to handle streamable backend
- API: Ok let me check feature flag if I can serve you streamable export
- SDK Receives streamable response: uses ExportTask internal (with is_export_v2=True) and tweaks its public API so it behaves just like Task class (result_url/errors_url)
Prepares export_v2 methods to use streamable backend under the hood