[@card] Realtime cards (3/N) #1552

valayDave · 2023-09-26T22:22:46Z

[card/components] internal `MetaflowCard` classes support realtime behavior

Markdown and Artifact Component support real-time updates.
default/blank cards support real-time update
Vega Charts And Progress Bar support in Python Land

TODO

Image Update support
[ ]

romain-intel

Minor nit but I think LGTM.

metaflow/plugins/cards/card_modules/basic.py

valayDave · 2023-11-26T20:43:22Z

metaflow/plugins/cards/card_modules/components.py

+    disable_updates: bool, optional
+        A boolean value to disable realtime updates for all components within the table. Default: False


Surface level API change to disable table updates when we want. cc @tuulos

do you have a particular use case in mind for this Table-level disable_updates?

Will Remove docstring cc @tuulos ; So that it's not a part of the public API

metaflow/plugins/cards/card_modules/components.py

valayDave · 2023-11-26T20:44:48Z

metaflow/plugins/cards/card_modules/components.py

+    def __init__(self, max=100, label=None, value=0, unit=None, metadata=None):
+        self._label = label
+        self._max = max
+        self._value = value
+        self._unit = unit
+        self._metadata = metadata
+
+    def update(self, new_value, metadata=None):
+        self._value = new_value
+        if metadata is not None:
+            self._metadata = metadata
+


User API signature needs validation : cc @tuulos

what's metadata?

metadata is any additional piece of information you want to add in the progressbar:

native_progress_bar = ProgressBar( max=sleep_cycles, label="Native Progress Bar", value=0, metadata="0 iter/s", )

valayDave · 2023-11-26T20:45:07Z

metaflow/plugins/cards/card_modules/components.py

+    def __init__(self, spec, data=None):
+        self._spec = spec
+        self._data = data
+
+    def update(self, data=None, spec=None):
+        if spec is not None:
+            self._spec = spec
+        if data is not None:
+            self._data = data
+
+    @classmethod
+    def from_altair_chart(cls, altair_chart):


User API signature needs validation : cc @tuulos

if I create a chart with from_altair_chart initially, do I then update the data with update(spec=None, data=my_data) or how would it work?

Yes I think you should be able to as long as the data object is the "full object"; The VegaChart object needs the full data object in the data update.

tuulos · 2023-12-03T22:40:05Z

metaflow/plugins/cards/card_modules/components.py

+        self._spec = spec
+        self._data = data
+
+    def update(self, data=None, spec=None):


spec is a JSON (a Python dict?) following the VegaLite spec here, right?

valayDave · 2023-12-04T10:06:38Z

metaflow/plugins/cards/card_modules/components.py

@@ -82,20 +82,28 @@ class Artifact(UserComponent):
        Use a truncated representation.
    """

+    REALTIME_UPDATABLE = True
+
+    def update(self, artifact):


valayDave · 2023-12-05T03:46:27Z

metaflow/plugins/cards/card_modules/components.py

+        self._spec = spec
+        self._data = data
+
+    def update(self, data=None, spec=None):


valayDave · 2023-12-08T06:59:36Z

metaflow/plugins/cards/card_modules/components.py

+    def __init__(self, max=100, label=None, value=0, unit=None, metadata=None):
+        self._label = label
+        self._max = max
+        self._value = value
+        self._unit = unit
+        self._metadata = metadata
+
+    def update(self, new_value, metadata=None):
+        self._value = new_value
+        if metadata is not None:
+            self._metadata = metadata
+


metadata is any additional piece of information you want to add in the progressbar:

native_progress_bar = ProgressBar( max=sleep_cycles, label="Native Progress Bar", value=0, metadata="0 iter/s", )

valayDave · 2023-12-08T07:00:25Z

metaflow/plugins/cards/card_modules/components.py

+    def __init__(self, spec, data=None):
+        self._spec = spec
+        self._data = data
+
+    def update(self, data=None, spec=None):
+        if spec is not None:
+            self._spec = spec
+        if data is not None:
+            self._data = data
+
+    @classmethod
+    def from_altair_chart(cls, altair_chart):


Yes I think you should be able to as long as the data object is the "full object"; The VegaChart object needs the full data object in the data update.

metaflow/plugins/cards/card_modules/components.py

valayDave · 2023-12-08T07:01:22Z

metaflow/plugins/cards/card_modules/components.py

+    disable_updates: bool
+        Disable realtime updates for the image. Default: True


Will Remove docstring cc @tuulos ; So that it's not a part of the public API

valayDave · 2023-12-08T07:01:40Z

metaflow/plugins/cards/card_modules/components.py

+    disable_updates: bool, optional
+        A boolean value to disable realtime updates for all components within the table. Default: False


Will Remove docstring cc @tuulos ; So that it's not a part of the public API

…vior - Markdown and Artifact Component made realtime. - default/blank cards made realtime - default component ids to realtime-updatable UserComponents

- Add a variable in the HTML template to enable/disable `metaflow_card_update` function

- Set options that allows disabling realtime updates for components that may have lots of data like `Image` / `Table`. This can be useful when we are calling refresh often and don't want to update data which maybe static.

- Image.update supports multi-type object - remove `disable_updates` from public api

- property in vega chart to show controls - remove `data` argument from Vega Chart. - It only takes `spec` as an argument. - VegaChart default width set to `container`

romain-intel

Haven't looked at things in great detail but I plan to test everything once it is all merged over the next week. My only big comment is that we should add docstrings/typehints to anything the user will use.

romain-intel · 2024-01-09T08:51:07Z

metaflow/plugins/cards/card_modules/components.py

+
+    REALTIME_UPDATABLE = True
+
+    def __init__(self, max=100, label=None, value=0, unit=None, metadata=None):


I'm going to be a stickler and say we should add type hints and docs :)

* [card-server] cli command to expose a card server - Server helps view realtime updates - Added a card viewer html file - Created a simple HTTP based card server that will help showcase the realtime cards from querying the server - The API response contains the task status for the data api - reload on run change : A background thread checks if there are new runs created via the latest_run file. - pass down poll_interval to card viewer - card server is also resiliant to no runs - You can set `METAFLOW_DEBUG_CARD_SERVER` to debug card server logs, Logging is off by default - clean url parsing for requests : needed to ensure calls to card server are resiliant to query params in requests. - also need clean url parsing for setting `embed` query param in the card iframe - run-info API sends task finished information which is useful when switching cards automatically when new tasks start. - it Always logs when run-ids change * [card-viewer-html] UI + Structure - Cleanup HTML template - Handles edge case where no runs are available - Shows errors when things are not working - leverages a configurable poll interval - set `embed` query param in the card iframe source. - it automatically switches to new card when tasks finish and other tasks are running. - Fix bug when card data is not available - checks for `final` reload token to stop polling updates * [@card] [card-core] always call `refresh` after first `render_runtime` (5/N) (#1670) * [card-core] always call `refresh` after first `render_runtime` - Added a `sync` argument to the card creator [@card][bug fixes] Address comments and squashing bugs (6/N) (#1678) * [card-components] type-hints + doc string Fix. * [card-core] fix function signature in tests. * [card-server] address comments on debug mode and hygiene * [card-components] vega charts special case inside table --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com>

* [card-components] internal MetaflowCard classes support realtime behavior - Markdown and Artifact Component made realtime. - default/blank cards made realtime - default component ids to realtime-updatable UserComponents * [default-card-python] add runtime check - Add a variable in the HTML template to enable/disable `metaflow_card_update` function * [card-components] progressbar / vegacharts * [card-components] customize updates for Tables/Images - Set options that allows disabling realtime updates for components that may have lots of data like `Image` / `Table`. This can be useful when we are calling refresh often and don't want to update data which maybe static. * [card-components] address comments on Image class - Image.update supports multi-type object - remove `disable_updates` from public api * [card-components][vega-chart][bug-fix] - property in vega chart to show controls - remove `data` argument from Vega Chart. - It only takes `spec` as an argument. - VegaChart default width set to `container` * [@card] Realtime cards (4/N) (#1553) * [card-server] cli command to expose a card server - Server helps view realtime updates - Added a card viewer html file - Created a simple HTTP based card server that will help showcase the realtime cards from querying the server - The API response contains the task status for the data api - reload on run change : A background thread checks if there are new runs created via the latest_run file. - pass down poll_interval to card viewer - card server is also resiliant to no runs - You can set `METAFLOW_DEBUG_CARD_SERVER` to debug card server logs, Logging is off by default - clean url parsing for requests : needed to ensure calls to card server are resiliant to query params in requests. - also need clean url parsing for setting `embed` query param in the card iframe - run-info API sends task finished information which is useful when switching cards automatically when new tasks start. - it Always logs when run-ids change * [card-viewer-html] UI + Structure - Cleanup HTML template - Handles edge case where no runs are available - Shows errors when things are not working - leverages a configurable poll interval - set `embed` query param in the card iframe source. - it automatically switches to new card when tasks finish and other tasks are running. - Fix bug when card data is not available - checks for `final` reload token to stop polling updates * [@card] [card-core] always call `refresh` after first `render_runtime` (5/N) (#1670) * [card-core] always call `refresh` after first `render_runtime` - Added a `sync` argument to the card creator [@card][bug fixes] Address comments and squashing bugs (6/N) (#1678) * [card-components] type-hints + doc string Fix. * [card-core] fix function signature in tests. * [card-server] address comments on debug mode and hygiene * [card-components] vega charts special case inside table --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com> --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com>

* [default-card][ui] default card realtime updatable - cleanup the ui component code and add naive progress-bar * [default-card][ui] Add custom progress bar * [default-card][ui] vegachart component - Chart support within tables and anywhere across the page - Chart mutation done based on spec/data changes. * [default-card][ui] remove chart.js deps. * [default-card][ui] migrate to vite * [default-card][ui] update progress bar styles * [default-card][ui] Fixes to chart/css/progess bar - pass down options in vega chart - progress-bar min width for mf UI - Fix bug when vega-chart spec changes. - fix `embed` class CSS * [@card] Realtime cards (3/N) (#1552) * [card-components] internal MetaflowCard classes support realtime behavior - Markdown and Artifact Component made realtime. - default/blank cards made realtime - default component ids to realtime-updatable UserComponents * [default-card-python] add runtime check - Add a variable in the HTML template to enable/disable `metaflow_card_update` function * [card-components] progressbar / vegacharts * [card-components] customize updates for Tables/Images - Set options that allows disabling realtime updates for components that may have lots of data like `Image` / `Table`. This can be useful when we are calling refresh often and don't want to update data which maybe static. * [card-components] address comments on Image class - Image.update supports multi-type object - remove `disable_updates` from public api * [card-components][vega-chart][bug-fix] - property in vega chart to show controls - remove `data` argument from Vega Chart. - It only takes `spec` as an argument. - VegaChart default width set to `container` * [@card] Realtime cards (4/N) (#1553) * [card-server] cli command to expose a card server - Server helps view realtime updates - Added a card viewer html file - Created a simple HTTP based card server that will help showcase the realtime cards from querying the server - The API response contains the task status for the data api - reload on run change : A background thread checks if there are new runs created via the latest_run file. - pass down poll_interval to card viewer - card server is also resiliant to no runs - You can set `METAFLOW_DEBUG_CARD_SERVER` to debug card server logs, Logging is off by default - clean url parsing for requests : needed to ensure calls to card server are resiliant to query params in requests. - also need clean url parsing for setting `embed` query param in the card iframe - run-info API sends task finished information which is useful when switching cards automatically when new tasks start. - it Always logs when run-ids change * [card-viewer-html] UI + Structure - Cleanup HTML template - Handles edge case where no runs are available - Shows errors when things are not working - leverages a configurable poll interval - set `embed` query param in the card iframe source. - it automatically switches to new card when tasks finish and other tasks are running. - Fix bug when card data is not available - checks for `final` reload token to stop polling updates * [@card] [card-core] always call `refresh` after first `render_runtime` (5/N) (#1670) * [card-core] always call `refresh` after first `render_runtime` - Added a `sync` argument to the card creator [@card][bug fixes] Address comments and squashing bugs (6/N) (#1678) * [card-components] type-hints + doc string Fix. * [card-core] fix function signature in tests. * [card-server] address comments on debug mode and hygiene * [card-components] vega charts special case inside table --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com> --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com> --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com>

* [default-card][ui] default card realtime updatable - cleanup the ui component code and add naive progress-bar * [default-card][ui] Add custom progress bar * [default-card][ui] vegachart component - Chart support within tables and anywhere across the page - Chart mutation done based on spec/data changes. * [default-card][ui] remove chart.js deps. * [default-card][ui] migrate to vite * [default-card][ui] update progress bar styles * [default-card][ui] Fixes to chart/css/progess bar - pass down options in vega chart - progress-bar min width for mf UI - Fix bug when vega-chart spec changes. - fix `embed` class CSS * [@card] Realtime cards (3/N) (Netflix#1552) * [card-components] internal MetaflowCard classes support realtime behavior - Markdown and Artifact Component made realtime. - default/blank cards made realtime - default component ids to realtime-updatable UserComponents * [default-card-python] add runtime check - Add a variable in the HTML template to enable/disable `metaflow_card_update` function * [card-components] progressbar / vegacharts * [card-components] customize updates for Tables/Images - Set options that allows disabling realtime updates for components that may have lots of data like `Image` / `Table`. This can be useful when we are calling refresh often and don't want to update data which maybe static. * [card-components] address comments on Image class - Image.update supports multi-type object - remove `disable_updates` from public api * [card-components][vega-chart][bug-fix] - property in vega chart to show controls - remove `data` argument from Vega Chart. - It only takes `spec` as an argument. - VegaChart default width set to `container` * [@card] Realtime cards (4/N) (Netflix#1553) * [card-server] cli command to expose a card server - Server helps view realtime updates - Added a card viewer html file - Created a simple HTTP based card server that will help showcase the realtime cards from querying the server - The API response contains the task status for the data api - reload on run change : A background thread checks if there are new runs created via the latest_run file. - pass down poll_interval to card viewer - card server is also resiliant to no runs - You can set `METAFLOW_DEBUG_CARD_SERVER` to debug card server logs, Logging is off by default - clean url parsing for requests : needed to ensure calls to card server are resiliant to query params in requests. - also need clean url parsing for setting `embed` query param in the card iframe - run-info API sends task finished information which is useful when switching cards automatically when new tasks start. - it Always logs when run-ids change * [card-viewer-html] UI + Structure - Cleanup HTML template - Handles edge case where no runs are available - Shows errors when things are not working - leverages a configurable poll interval - set `embed` query param in the card iframe source. - it automatically switches to new card when tasks finish and other tasks are running. - Fix bug when card data is not available - checks for `final` reload token to stop polling updates * [@card] [card-core] always call `refresh` after first `render_runtime` (5/N) (Netflix#1670) * [card-core] always call `refresh` after first `render_runtime` - Added a `sync` argument to the card creator [@card][bug fixes] Address comments and squashing bugs (6/N) (Netflix#1678) * [card-components] type-hints + doc string Fix. * [card-core] fix function signature in tests. * [card-server] address comments on debug mode and hygiene * [card-components] vega charts special case inside table --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com> --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com> --------- Co-authored-by: adam <203779+seethroughdev@users.noreply.github.com>

valayDave changed the title ~~[card/components] internal MetaflowCard classes support realtime behavior~~ [@card] Realtime cards (3/N) Sep 26, 2023

valayDave mentioned this pull request Sep 26, 2023

[@card] Realtime cards (1/N) #1550

Merged

2 tasks

valayDave marked this pull request as ready for review September 26, 2023 22:48

valayDave requested review from tuulos, romain-intel and saikonen September 26, 2023 23:01

valayDave force-pushed the valay/rtc-stack-1 branch from 9a27a2b to eba36be Compare September 26, 2023 23:22

valayDave force-pushed the valay/rtc-stack-2 branch from c2ed928 to 66d3bea Compare September 26, 2023 23:22

romain-intel reviewed Oct 3, 2023

View reviewed changes

metaflow/plugins/cards/card_modules/basic.py Outdated Show resolved Hide resolved

valayDave force-pushed the valay/rtc-stack-1 branch from eba36be to 4bb9032 Compare October 11, 2023 18:56

valayDave force-pushed the valay/rtc-stack-2 branch from 66d3bea to 5e05ed6 Compare October 11, 2023 18:56

valayDave force-pushed the valay/rtc-stack-1 branch from 4bb9032 to e5e870f Compare October 18, 2023 01:49

valayDave force-pushed the valay/rtc-stack-2 branch from 5e05ed6 to 4816fb0 Compare October 18, 2023 01:49

valayDave commented Oct 18, 2023

View reviewed changes

metaflow/plugins/cards/card_modules/basic.py Outdated Show resolved Hide resolved

valayDave force-pushed the valay/rtc-stack-2 branch from 4816fb0 to eb56ecb Compare October 18, 2023 01:55

valayDave force-pushed the valay/rtc-stack-1 branch from e5e870f to d5f8c75 Compare October 18, 2023 17:10

valayDave force-pushed the valay/rtc-stack-2 branch from eb56ecb to bb156d6 Compare October 18, 2023 17:10

valayDave force-pushed the valay/rtc-stack-1 branch from d5f8c75 to fa23ad4 Compare October 18, 2023 17:24

valayDave force-pushed the valay/rtc-stack-2 branch from bb156d6 to 505c7ab Compare October 18, 2023 17:24

valayDave force-pushed the valay/rtc-stack-1 branch from fa23ad4 to 272346a Compare October 19, 2023 21:48

valayDave force-pushed the valay/rtc-stack-2 branch from 505c7ab to 48f4936 Compare October 19, 2023 21:48

valayDave force-pushed the valay/rtc-stack-1 branch from 272346a to 57e3a7e Compare October 31, 2023 19:36

valayDave force-pushed the valay/rtc-stack-2 branch from 48f4936 to d100a35 Compare October 31, 2023 19:36

valayDave force-pushed the valay/rtc-stack-2 branch from d100a35 to 7502704 Compare November 19, 2023 20:53

valayDave changed the base branch from valay/rtc-stack-1 to master November 19, 2023 21:06

valayDave changed the base branch from master to valay/rtc-stack-1 November 19, 2023 21:07

valayDave force-pushed the valay/rtc-stack-1 branch from 087fe3e to 2a8501a Compare November 21, 2023 06:40

valayDave force-pushed the valay/rtc-stack-2 branch from 28b49f3 to 04930e7 Compare November 21, 2023 06:40

valayDave force-pushed the valay/rtc-stack-1 branch from 2a8501a to be176e3 Compare November 21, 2023 07:03

valayDave force-pushed the valay/rtc-stack-2 branch from 04930e7 to 4ce5958 Compare November 21, 2023 07:03

valayDave commented Nov 26, 2023

View reviewed changes

tuulos reviewed Dec 3, 2023

View reviewed changes

valayDave force-pushed the valay/rtc-stack-1 branch from be176e3 to 78c7154 Compare December 6, 2023 04:56

valayDave force-pushed the valay/rtc-stack-2 branch from 4ce5958 to c0e510a Compare December 6, 2023 04:56

valayDave changed the base branch from valay/rtc-stack-1 to master December 6, 2023 05:03

valayDave changed the base branch from master to valay/rtc-stack-1 December 6, 2023 05:03

valayDave commented Dec 8, 2023

View reviewed changes

valayDave force-pushed the valay/rtc-stack-1 branch from b55df99 to 93329b3 Compare January 8, 2024 14:38

valayDave force-pushed the valay/rtc-stack-2 branch from 31f714b to 8158bd8 Compare January 8, 2024 14:38

valayDave added 6 commits January 8, 2024 06:45

[card-components] internal MetaflowCard classes support realtime beha…

585fbb2

…vior - Markdown and Artifact Component made realtime. - default/blank cards made realtime - default component ids to realtime-updatable UserComponents

[default-card-python] add runtime check

d2b2f76

- Add a variable in the HTML template to enable/disable `metaflow_card_update` function

[card-components] progressbar / vegacharts

eacab45

[card-components] customize updates for Tables/Images

8760cc8

- Set options that allows disabling realtime updates for components that may have lots of data like `Image` / `Table`. This can be useful when we are calling refresh often and don't want to update data which maybe static.

[card-components] address comments on Image class

7caf7fd

- Image.update supports multi-type object - remove `disable_updates` from public api

[card-components][vega-chart][bug-fix]

cf28b1f

- property in vega chart to show controls - remove `data` argument from Vega Chart. - It only takes `spec` as an argument. - VegaChart default width set to `container`

valayDave force-pushed the valay/rtc-stack-2 branch from 3e7e4a0 to cf28b1f Compare January 8, 2024 14:47

valayDave force-pushed the valay/rtc-stack-1 branch from 615ffe4 to 947c2ec Compare January 8, 2024 14:47

romain-intel approved these changes Jan 9, 2024

View reviewed changes

valayDave merged commit fc430d4 into valay/rtc-stack-1 Jan 11, 2024
1 check passed

valayDave deleted the valay/rtc-stack-2 branch January 11, 2024 16:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[@card] Realtime cards (3/N) #1552

[@card] Realtime cards (3/N) #1552

valayDave commented Sep 26, 2023 •

edited

romain-intel left a comment

valayDave Nov 26, 2023

tuulos Dec 3, 2023

valayDave Dec 8, 2023

valayDave Nov 26, 2023

tuulos Dec 3, 2023

valayDave Dec 8, 2023

valayDave Nov 26, 2023

tuulos Dec 3, 2023

valayDave Dec 8, 2023

tuulos Dec 3, 2023

valayDave Dec 5, 2023

valayDave Dec 4, 2023

valayDave Dec 5, 2023

valayDave Dec 8, 2023

valayDave Dec 8, 2023

valayDave Dec 8, 2023

valayDave Dec 8, 2023

romain-intel left a comment

romain-intel Jan 9, 2024

		disable_updates: bool, optional
		A boolean value to disable realtime updates for all components within the table. Default: False

		disable_updates: bool
		Disable realtime updates for the image. Default: True


		REALTIME_UPDATABLE = True

		def __init__(self, max=100, label=None, value=0, unit=None, metadata=None):

[@card] Realtime cards (3/N) #1552

[@card] Realtime cards (3/N) #1552

Conversation

valayDave commented Sep 26, 2023 • edited

[card/components] internal MetaflowCard classes support realtime behavior

TODO

romain-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romain-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valayDave commented Sep 26, 2023 •

edited

[card/components] internal `MetaflowCard` classes support realtime behavior