Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Maintaining aggregation state between task calls #6

Closed
wesm opened this issue Apr 21, 2015 · 2 comments
Closed

Maintaining aggregation state between task calls #6

wesm opened this issue Apr 21, 2015 · 2 comments

Comments

@wesm
Copy link
Member

wesm commented Apr 21, 2015

Issue by wesm
Tuesday Dec 09, 2014 at 20:08 GMT
Originally opened as http://github.mtv.cloudera.com/wesm/ibis/issues/6

In order to reduce serialization / data transfer costs between consecutive Update tasks. In the case of a group-wise aggregation, many aggregation states might exist simultaneously in a particular Python worker process. These can be identified by a UUID and the UUID will have to be made known to the parent Impala process

One problem with this is that we would have to keep sending tasks to the same worker process, so releasing the worker back to the pool could become problematic.
maxmzkr added a commit to maxmzkr/ibis that referenced this issue Nov 22, 2016
@maxmzkr
Merge pull request ibis-project#6 from maxmzkr/user-needed-in-cursor
b9aa053 
User needed in cursor
@cpcloud
Copy link
Member

cpcloud commented Jun 28, 2017

@wesm Is this issue still relevant?

@wesm
Copy link
Member Author

wesm commented Jun 28, 2017

Nope

@wesm wesm closed this as completed Jun 28, 2017
anmyachev added a commit to anmyachev/ibis that referenced this issue Mar 26, 2020
@anmyachev @jrbourbeau
@semelianova @xmnlab @LeeTZ @datapythonista
merge ibis_project/ibis master (ibis-project#6)
890ccd1 
* Add initial support for ibis.random function (ibis-project#2060)

* remove drop method (ibis-project#2114)

* FEAT: OmnisciDB Supporting of cumulative ops through impala (ibis-project#2113)

* FEAT: OmniSciDB - Add support for within, d_fully_within and point (ibis-project#2125)

* [OmniSciDB] Fix TopK when used as filter (ibis-project#2134)

* FEAT: Implement read_csv for omniscidb backend (ibis-project#2062)

* FEAT: add translation rules for isnull() and notnull() for pyspark backend (ibis-project#2126)

* Add non-nullable info to schema output. (ibis-project#2117)

* CI: Fix PYTHONHASHSEED issue (ibis-project#2140)

* FEAT: OmniSciDB Added fragment_size to table creation (ibis-project#2107)

* TST: OmniSciDB - Enable logical value tests (ibis-project#2084)

* CLN: Rename tutorials to keep order (ibis-project#2145)

* FEAT: OmniSciDB ops.Date, ops.DateDiff and ops.TimestampDiff implementation (ibis-project#2097)

Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>
Co-authored-by: Svetlana Emelianova <44813696+semelianova@users.noreply.github.com>
Co-authored-by: Ivan Ogasawara <ivan.ogasawara@gmail.com>
Co-authored-by: Tiezheng Li <litiezheng513@gmail.com>
Co-authored-by: Marc Garcia <garcia.marc@gmail.com>
amyskov pushed a commit to amyskov/ibis that referenced this issue Mar 31, 2020
@semelianova
Merge pull request ibis-project#6 from semelianova/fix
97c256e 
Some fixes
lostmygithubaccount added a commit to lostmygithubaccount/ibis that referenced this issue Aug 12, 2023
@lostmygithubaccount
# This is a combination of 10 commits.
a520a4f 
# This is the 1st commit message:

fix justfile

# This is the commit message ibis-project#2:

update ai app

# This is the commit message ibis-project#3:

fix posts

# This is the commit message ibis-project#4:

add very minimal README

# This is the commit message ibis-project#5:

move to dev quarto; fix homepage; remove ai link

# This is the commit message ibis-project#6:

qs

# This is the commit message ibis-project#7:

qs

# This is the commit message ibis-project#8:

qs

# This is the commit message ibis-project#9:

qs

# This is the commit message ibis-project#10:

qs
deepyaman added a commit to deepyaman/ibis that referenced this issue Oct 30, 2023
@deepyaman
# This is a combination of 7 commits.
46d03af 
# This is the 1st commit message:

build: format code with `ruff format` over `black`

# This is the commit message ibis-project#2:

docs: fix dask compile docstring typo
# This is the commit message ibis-project#3:

fix(datafusion): fix some temporal operations

# This is the commit message ibis-project#4:

feat(datafusion): add case and if-else statements

# This is the commit message ibis-project#5:

chore(flake/nixpkgs): `0e18a0a0` -> `37c290f0`

# This is the commit message ibis-project#6:

chore(deps): update trinodb/trino docker tag to v431
# This is the commit message ibis-project#7:

chore(flake/poetry2nix): `3b6124e9` -> `365879d4`
deepyaman added a commit to deepyaman/ibis that referenced this issue Oct 30, 2023
@deepyaman
# This is a combination of 7 commits.
d3dd4d2 
# This is the 1st commit message:

build: format code with `ruff format` over `black`

# This is the commit message ibis-project#2:

docs: fix dask compile docstring typo
# This is the commit message ibis-project#3:

fix(datafusion): fix some temporal operations

# This is the commit message ibis-project#4:

feat(datafusion): add case and if-else statements

# This is the commit message ibis-project#5:

chore(flake/nixpkgs): `0e18a0a0` -> `37c290f0`

# This is the commit message ibis-project#6:

chore(deps): update trinodb/trino docker tag to v431
# This is the commit message ibis-project#7:

chore(flake/poetry2nix): `3b6124e9` -> `365879d4`
deepyaman added a commit to deepyaman/ibis that referenced this issue Oct 30, 2023
@deepyaman
# This is a combination of 14 commits.
fd4f7dc 
# This is the 1st commit message:

test(flink): turn on the common backend test suite

Co-authored-by: mfatihaktas <mfatihaktas@gmail.com>

# This is the commit message ibis-project#2:

chore(flink): improve marker re deserializing JSON

# This is the commit message ibis-project#3:

test(flink): summarize the xfail reason for a test

# This is the commit message ibis-project#4:

refactor(flink): condense logic for table creation

# This is the commit message ibis-project#5:

chore(flink): mark WIP tests using notimpl instead

# This is the commit message ibis-project#6:

chore(flink): remove the useless/redundant reasons

# This is the commit message ibis-project#7:

chore(flink): change a "broken" `mark` to "notyet"

# This is the commit message ibis-project#8:

chore(flink): remove backend-specific expectations

# This is the commit message ibis-project#9:

test(flink): remove conditional expected_for_flink

# This is the commit message ibis-project#10:

chore(flake/nixpkgs): `0e18a0a0` -> `37c290f0`

# This is the commit message ibis-project#11:

chore(deps): update trinodb/trino docker tag to v431
# This is the commit message ibis-project#12:

chore(flake/poetry2nix): `3b6124e9` -> `365879d4`

# This is the commit message ibis-project#13:

chore: constrain using expanded bounds syntax

# This is the commit message ibis-project#14:

chore(flake/nixpkgs): `37c290f0` -> `a9d001fd`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wesm @cpcloud