Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve serialization of Pandas DataFrames to ipyvega #346

Merged
merged 164 commits into from
Feb 12, 2023
Merged
Show file tree
Hide file tree
Changes from 158 commits
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
01aab0d
data transfer via ipydatawidgets
xtianpoli Jun 7, 2021
162a38e
fix
xtianpoli Jun 8, 2021
d1989f7
Fix heatmap for speed
jdfekete Jun 11, 2021
8813074
Fix heatmap for speed
jdfekete Jun 11, 2021
28ba8e9
init ipydatatablewidget
xtianpoli Jun 11, 2021
44efd02
using NDArray instead of DataUnion
xtianpoli Jun 13, 2021
81e90a4
adding a customized table traitlet
xtianpoli Jun 17, 2021
94c7077
src/widget.ts
xtianpoli Jun 17, 2021
de25ee6
fix
xtianpoli Jun 17, 2021
69c689f
cleanup
xtianpoli Jun 17, 2021
88e5138
adding lz4js decompression
xtianpoli Jun 18, 2021
c2f33c9
fix
xtianpoli Jun 18, 2021
5d58cba
make NumpyAdapter.equals returns always False
xtianpoli Jun 18, 2021
21e9bea
Remove most console.log and add %time and progress bars in notebooks
jdfekete Jun 18, 2021
803a0cb
fixing .gitignore
xtianpoli Jun 18, 2021
3badb98
Add stress tests
jdfekete Jun 18, 2021
b198c04
rename update_histogram2d to update_array2d. Remove console.log pollu…
jdfekete Jun 19, 2021
62eaa47
Improve
jdfekete Jun 21, 2021
aa0b328
Add a definition file to configure Altair. More work needed
jdfekete Jun 21, 2021
c856d7a
compression as string
xtianpoli Jun 28, 2021
258e0aa
fix (naive approach)
xtianpoli Jun 29, 2021
0d6c5d3
adding an adapter for Progressivis tables
xtianpoli Jun 29, 2021
5498083
Merge branch 'vega:master' into master
xtianpoli Jun 29, 2021
a814023
cleanup
jdfekete Jun 29, 2021
283fe75
adding a touch mode
xtianpoli Jul 1, 2021
1941c41
using ipytablewidgets
xtianpoli Jul 12, 2021
ca21ed7
calling update_dataframe() and update_array2d() from update()
xtianpoli Aug 24, 2021
6601ff1
fixes for jupyterlab3 compliance
xtianpoli Aug 25, 2021
3b128ce
adding UI tests
xtianpoli Aug 27, 2021
0b400c6
adding reference outputs for UI tests
xtianpoli Aug 27, 2021
162784f
add ui action
xtianpoli Aug 27, 2021
ad5acc9
fix syntax
xtianpoli Aug 27, 2021
94828b6
another fix
xtianpoli Aug 27, 2021
d672ca7
fix poetry conf
xtianpoli Aug 27, 2021
1e2842b
testing only with python 3.9
xtianpoli Aug 27, 2021
64246db
testing with yarn --no-lockfile
xtianpoli Aug 27, 2021
19ea342
testing only on lab branch for now
xtianpoli Aug 27, 2021
af41998
trusting test notebook
xtianpoli Aug 27, 2021
7499f3f
debug: testing ipytablewidgets presence
xtianpoli Aug 27, 2021
40fd5a6
get a cell as an artifact
xtianpoli Aug 27, 2021
eac30ba
fix syntax
xtianpoli Aug 27, 2021
724129c
if: always()
xtianpoli Aug 27, 2021
b69674b
arfifact test-output instead of ref...
xtianpoli Aug 27, 2021
9fd014a
uploading all .png as an artifact
xtianpoli Aug 27, 2021
c4a8b67
new refs
xtianpoli Aug 27, 2021
cc33cd0
upload artifacts only if: cancelled() to avoid warnings
xtianpoli Aug 27, 2021
fff5fe6
improvement
xtianpoli Aug 30, 2021
defb652
Widget.ipynb modified for use datasets property
xtianpoli Aug 31, 2021
9bb7a22
adding a random.seed()
xtianpoli Aug 31, 2021
9c6ad58
fixes
xtianpoli Sep 1, 2021
e82f971
more ui tests
xtianpoli Sep 1, 2021
534113c
fix: producing artifacts if failure
xtianpoli Sep 1, 2021
4f8250b
fix
xtianpoli Sep 1, 2021
de46fe8
adding new ref
xtianpoli Sep 1, 2021
87a0441
simplifying last tests
xtianpoli Sep 1, 2021
bf4d1ba
resize histogram2d to 32x32
xtianpoli Sep 1, 2021
989d034
adding new refernces
xtianpoli Sep 1, 2021
ddfc9d7
many updates ported from the bench branch
xtianpoli Oct 1, 2021
0fbf69e
add jupyterlab
jdfekete Oct 5, 2021
2d64f00
Introduce vega.altair with the `stream` method to stream altair specs
jdfekete Oct 11, 2021
be82d76
fix bugs and add interactive test
jdfekete Oct 13, 2021
c1dd5a4
Call resize after update in streaming mode, otherwise, the widget is …
jdfekete Oct 15, 2021
89a0a44
Add the resize option to stream and update
jdfekete Oct 16, 2021
e4ff083
fixing resize for pending updates
xtianpoli Oct 18, 2021
9ec8cae
Fix tests
jdfekete Nov 7, 2021
86d885d
Simplify streaming test
jdfekete Nov 8, 2021
58bd0d0
Ready for distribution
jdfekete Nov 8, 2021
34ba868
fix test.yml
xtianpoli Nov 21, 2021
8d0fefe
prettier
jdfekete Nov 21, 2021
05f7bd9
merged with origin
jdfekete Nov 21, 2021
c5815bc
Cleanup
jdfekete Nov 21, 2021
d45d9ce
Fixed spec
jdfekete Nov 21, 2021
4eaba72
Merge branch 'master' into master
jdfekete Dec 7, 2021
320841a
bump to new library versions for yarn and vega
jdfekete Dec 7, 2021
e295597
fix unused packages
jdfekete Dec 7, 2021
5eff2f0
fixing ui tests
xtianpoli Jan 20, 2022
a24a73d
other fixes rlated to the 346 issue
xtianpoli Jan 21, 2022
ee68fe7
cleanup+fix
xtianpoli Jan 21, 2022
52672be
new fixes
xtianpoli Jan 21, 2022
026116d
set nodejs=17
xtianpoli Jan 21, 2022
9aed873
set nodejs=16
xtianpoli Jan 21, 2022
e7a2812
make variables that can be const const
xtianpoli Jan 22, 2022
5e80acf
cleanup
xtianpoli Jan 22, 2022
248bf6d
Add comment for chaining multiple datasets in widget.py. Cleanup decl…
jdfekete Jan 22, 2022
c196c6e
use the latest version of Altair. Print better information in altair.…
jdfekete Jan 23, 2022
127df68
Try lower version of Altair
jdfekete Jan 23, 2022
1dd29cd
trying altair4.1.0+jsonschema3.2.0
xtianpoli Jan 23, 2022
a9615e9
fixing the previous essay
xtianpoli Jan 23, 2022
a65419f
altair back to dev-dependencies
xtianpoli Jan 23, 2022
af4f9c7
alt.renderers.enable('default') instead of 'notebook'
xtianpoli Jan 23, 2022
755ae8a
Add a first cell in AltairStreaming.ipynb to explain to install Altai…
jdfekete Jan 23, 2022
3da3644
UI end-to-end tests without conda
xtianpoli Jan 27, 2022
cd98d68
trying sleep 3
xtianpoli Jan 27, 2022
d021a09
trying py 3.8
xtianpoli Jan 27, 2022
7cfb6b4
py 3.7 ?
xtianpoli Jan 27, 2022
736e035
py 3.6 ?
xtianpoli Jan 27, 2022
9923598
avoid any[]|string
xtianpoli Jan 27, 2022
1d3f5fd
removing touch_mode and touch params
xtianpoli Jan 28, 2022
c3dc68f
removing binder conf
xtianpoli Jan 28, 2022
ca714b3
using a logger instead of print()
xtianpoli Jan 28, 2022
5dee6ff
insert?: any[] | @dataframe | ...
xtianpoli Jan 28, 2022
71e4b0b
clean-up code, add comments, check the need for resize in vega.wiget,…
jdfekete Jan 28, 2022
0c571f5
Altair should be either a dev dependency or a strong dependency, not …
jdfekete Jan 28, 2022
8443355
Add comments to the AltairStreaming.ipynb notebook [skip CI]
jdfekete Jan 29, 2022
8c9dcce
update dependencies
jdfekete Jan 31, 2022
acbba0c
update webpack
jdfekete Jan 31, 2022
f655f5d
Merge branch 'master' into master
jdfekete Jan 31, 2022
4c452b1
altair back to dev dependencies. Update webpack package version.
jdfekete Feb 2, 2022
2a5a051
Update webpack-cli package version.
jdfekete Feb 2, 2022
be9948e
Merge branch 'master' into master
jdfekete Feb 2, 2022
ec18ab7
fix yarn lockfile issue
jdfekete Feb 2, 2022
e4adc30
Apply requested changes, except for moving the dependencies of jupyte…
jdfekete Feb 3, 2022
708d01a
Bump to the newer version of Poetry
jdfekete Feb 9, 2022
b6bccbb
Update src/index.ts
jdfekete Feb 9, 2022
db964af
Rename AllSupportedTypes.ipynb to AllCompressions.ipynb and fix the R…
jdfekete Feb 9, 2022
7f3b1de
Use stable URL for example data
jdfekete Feb 9, 2022
2222e99
adding extension.js labplugin.js
xtianpoli Feb 10, 2022
014494e
removing TimeXXX.ipynb
xtianpoli Feb 10, 2022
09baecb
"adding jupyter nbextension enable" subcommand, removing comment "not…
xtianpoli Feb 10, 2022
991d9c0
Start merging with ipyvega base but the VegaWidget has issues with er…
jdfekete Feb 11, 2022
4ebd98f
Remove unused ISerializers from import in wiget.ts [skip CI]
jdfekete Feb 11, 2022
8868c6a
fixing bug in vega.js, removing src/extension.js (replaced by extensi…
xtianpoli Mar 11, 2022
b12606e
waiting 10s after start-jlab before running UI tests
xtianpoli Mar 11, 2022
3f43bfd
keeping only py3.9
xtianpoli Mar 11, 2022
8c55e03
prettifying .ts files + back to py3.6..3.9
xtianpoli Mar 11, 2022
3b63e9d
a better fix for src/vega.js
xtianpoli Mar 14, 2022
8a54ffc
Merge with upstream
jdfekete Mar 16, 2022
f2ffe37
Merge branch 'vega-master2'
jdfekete Mar 16, 2022
774c696
fix jupyterlab
jdfekete Mar 16, 2022
5fff98b
fix poetry.lock
jdfekete Mar 16, 2022
f57a22d
fix test
jdfekete Mar 16, 2022
9fb8c8e
new fix for test
jdfekete Mar 16, 2022
1b8a2ad
trying pip install for extra jupyterlab
xtianpoli Mar 16, 2022
3004e50
fix syntax error
xtianpoli Mar 16, 2022
3384981
trying poetry without virtualisation
xtianpoli Mar 16, 2022
bf009ae
ipytablewidgets=0.2.4, jupyterlab extra removed
xtianpoli Mar 17, 2022
eb0c920
fixes
xtianpoli Mar 17, 2022
7c75a82
adding forgotten .js
xtianpoli Mar 17, 2022
350d50d
fixes
xtianpoli Mar 17, 2022
4ed20f3
essay
xtianpoli Mar 17, 2022
420449c
many fixes
xtianpoli Mar 21, 2022
4c589bb
Merge branch 'master' into master
jdfekete Jun 23, 2022
b0f3144
adding jupyter-tablewidgets
xtianpoli Jun 23, 2022
14dc3f9
fix
xtianpoli Jun 23, 2022
033e78b
error injection for test
xtianpoli Jun 23, 2022
9b7ca01
fixing UI Tests
xtianpoli Jun 23, 2022
bec53bc
back to the previous fix
xtianpoli Jun 23, 2022
c6f6b43
save widgets rendering as images
xtianpoli Jan 10, 2023
e1b7a58
using poetry 1.3.1
xtianpoli Jan 10, 2023
82b6899
minor improvement
xtianpoli Jan 10, 2023
71fc3ef
serialize static widgets as json
xtianpoli Jan 11, 2023
d5d116b
Regenerate poetry lock file and fix warning in tests
jdfekete Jan 19, 2023
bfc96f6
Update packages and try saving a streamed example
jdfekete Jan 19, 2023
dacf4fb
setting rendered images dims
xtianpoli Jan 19, 2023
6e5fc40
back to @jupyter-widgets/base=4.1.0 because of jupyterlab
xtianpoli Jan 19, 2023
9a067f3
fix
xtianpoli Jan 19, 2023
b0493bb
new fixes+upgraded dependencies
xtianpoli Jan 20, 2023
e70423f
upgrade dev dependency filemanager-webpack-plugin => 7.0.0
xtianpoli Jan 21, 2023
c930fc8
merge upstream
xtianpoli Jan 30, 2023
baca3f0
rm .gitmodules+update poetry.loc
xtianpoli Jan 30, 2023
ba96399
using barley.json instead of barley.csv in all cells
xtianpoli Jan 30, 2023
3adb15a
improvements
xtianpoli Jan 30, 2023
10714a0
Improve documentation to function `stream`
jdfekete Jan 31, 2023
d618580
Fix docstring
jdfekete Jan 31, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:

strategy:
matrix:
python: ["3.6", "3.7", "3.8", "3.9"]
python: ["3.7", "3.8", "3.9", "3.10"]

name: Python ${{ matrix.python }}

Expand All @@ -29,7 +29,7 @@ jobs:
- name: Setup poetry
uses: abatilo/actions-poetry@v2.1.4
with:
poetry-version: 1.1.12
poetry-version: 1.3.1

- name: Configure poetry
run: poetry config virtualenvs.in-project true
Expand Down
68 changes: 68 additions & 0 deletions .github/workflows/ui.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
name: UI Tests

on:
push:
branches:
- master
pull_request:
branches:
- master

jobs:
run:
runs-on: ubuntu-latest

strategy:
matrix:
domoritz marked this conversation as resolved.
Show resolved Hide resolved
python: ["3.7", "3.8", "3.9", "3.10"]

name: Python ${{ matrix.python }}

defaults:
run:
shell: bash -l {0}

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python }}
uses: actions/setup-python@v2.3.1
with:
python-version: ${{ matrix.python }}

- name: Setup poetry
uses: abatilo/actions-poetry@v2.1.4
with:
poetry-version: 1.3.1

- name: Configure poetry
run: poetry config virtualenvs.create false

- name: Install Python dependencies
run: |
poetry install
python -m pip install jupyterlab-widgets==1.0.2
python -m pip install jupyterlab==3.0.11

- name: Setup Node
uses: actions/setup-node@v2.5.1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Install Node dependencies
run: yarn --frozen-lockfile

- name: Install Node dependencies
run: yarn --frozen-lockfile

- name: Build
run: yarn run build

- name: Run tests
run: |
jupyter labextension install .
cd ui-tests/
yarn
domoritz marked this conversation as resolved.
Show resolved Hide resolved
yarn run start-jlab:detached
sleep 10
yarn run test
- name: Upload output file
if: failure()
uses: actions/upload-artifact@v2
with:
name: cell-artifact
path: ui-tests/test-output/test/screenshots/vega_cell_*.png
Empty file added .gitmodules
Empty file.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ To install `vega` and its dependencies from the Python Package Index using
```sh
pip install jupyter pandas vega
pip install --upgrade notebook # need jupyter_client >= 4.2 for sys-prefix below
jupyter nbextension install --sys-prefix --py vega # not needed in notebook >= 5.3
jdfekete marked this conversation as resolved.
Show resolved Hide resolved
jupyter nbextension install --sys-prefix --py vega
jupyter nbextension enable --py --sys-prefix vega
```

### Conda Forge
Expand Down Expand Up @@ -51,7 +52,8 @@ Then activate the virtual environment with `poetry shell`.
Symlink files instead of copying files:

```sh
jupyter nbextension install --py --symlink vega
jupyter nbextension install --py --symlink --sys-prefix vega
jupyter nbextension enable --py --sys-prefix vega
```

Run kernel with `jupyter notebook`. Run the tests with `pytest vega`.
Expand Down
214 changes: 214 additions & 0 deletions notebooks/AllCompressions.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "247bdd49",
"metadata": {},
"source": [
"# Compression methods\n",
"Two compression methods are provided to send data: zlib and lz4. The latter is enabled by default since it is both fast and efficient for numerical columns. You may experience better results for particular column types so feel free to tune the compression method for your particular data.\n",
"\n",
"With a `VegaWidget`, the field `.compression` specifies the method used to send data.\n",
"The following code shows how to do it and some time results. Keep in mind that the time returned by Python are not exactly related to the performance of the compression because the data transmission is asynchronous. The numbers are still informative.\n",
"The content of the `.compression` field can be either a string with the name of the compression or an instance of the compressor, to specify the compression level."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "688a1886",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d23699478a5e48f98027c11ce7792fb7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VegaWidget()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 6 µs, sys: 7 µs, total: 13 µs\n",
"Wall time: 15.3 µs\n"
]
}
],
"source": [
"spec_no_data = {\n",
" \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.json\",\n",
" \"data\": {\"name\": \"data\"},\n",
" \"mark\": \"bar\",\n",
" \"encoding\": {\n",
" \"x\": {\"aggregate\": \"sum\", \"field\": \"yield\"},\n",
" \"y\": {\"field\": \"variety\"},\n",
" \"color\": {\"field\": \"site\"}\n",
" }\n",
"}\n",
"from vega.widget import VegaWidget\n",
"import requests\n",
"import json\n",
"req = requests.get(\"https://cdn.jsdelivr.net/npm/vega-datasets@2.2.0/data/barley.json\")\n",
"values = json.loads(req.text)\n",
"#data\n",
"widget = VegaWidget(spec=spec_no_data)\n",
"display(widget)\n",
"%time widget.update('data', insert=values)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "7d0f4bf0",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5864bd22268b4b03bfecb71af6206a23",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VegaWidget()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 7.41 ms, sys: 0 ns, total: 7.41 ms\n",
"Wall time: 7.1 ms\n"
]
}
],
"source": [
"import pandas as pd\n",
"URL = \"https://forge.scilab.org/index.php/p/rdataset/source/file/368b19abcb4292c56e4f21079f750eb76b325907/csv/lattice/barley.csv\"\n",
"df = pd.read_csv(URL)\n",
"widget = VegaWidget(spec=spec_no_data)\n",
"display(widget)\n",
"%time widget.update(\"data\", insert=df)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a13d013c",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "112e7a4059d64fa08cab2f0583de3074",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VegaWidget()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 9.44 ms, sys: 0 ns, total: 9.44 ms\n",
"Wall time: 8.62 ms\n"
]
}
],
"source": [
"import pandas as pd\n",
"URL = \"https://forge.scilab.org/index.php/p/rdataset/source/file/368b19abcb4292c56e4f21079f750eb76b325907/csv/lattice/barley.csv\"\n",
"df = pd.read_csv(URL)\n",
"widget = VegaWidget(spec=spec_no_data)\n",
"widget.compression = 'zlib'\n",
"display(widget)\n",
"%time widget.update('data', insert=df)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "6792044d",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ab67b74c7db74b1896e859031a0deb72",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VegaWidget()"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 9.84 ms, sys: 1.57 ms, total: 11.4 ms\n",
"Wall time: 10.7 ms\n"
]
}
],
"source": [
"import pandas as pd\n",
"from ipytablewidgets import LZ4Compressor\n",
"URL = \"https://forge.scilab.org/index.php/p/rdataset/source/file/368b19abcb4292c56e4f21079f750eb76b325907/csv/lattice/barley.csv\"\n",
"df = pd.read_csv(URL)\n",
"widget = VegaWidget(spec=spec_no_data)\n",
"widget.compression = LZ4Compressor(2)\n",
"display(widget)\n",
"%time widget.update('data', insert=df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11890337",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
109 changes: 57 additions & 52 deletions notebooks/Altair.ipynb

Large diffs are not rendered by default.

Loading