Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement a histogram operation type for numeric fields #33

Closed
wesm opened this issue Apr 21, 2015 · 1 comment
Closed

Implement a histogram operation type for numeric fields #33

wesm opened this issue Apr 21, 2015 · 1 comment
Labels
analytics Higher-level data analysis API related issues or PRs feature Features or general enhancements
Milestone
0.2

Comments

@wesm
Copy link
Member

wesm commented Apr 21, 2015

Issue by wesm
Tuesday Jan 06, 2015 at 18:07 GMT
Originally opened as http://github.mtv.cloudera.com/wesm/ibis/issues/33

This can exist as a first-class operator in the expression IR. For the Impala implementation, this will require an aggregation inline view (to get the min/max), a [cross] join (to make the min/max available to all rows), and an expression to get the histogram bucket number.

It might be interesting to enable group-wise histograms (a different bucketing for each group).
@wesm wesm added this to the 0.2 milestone Apr 21, 2015
@wesm wesm added the feature Features or general enhancements label Apr 21, 2015
@wesm
Copy link
Member Author

wesm commented Apr 21, 2015

Comment by wesm
Monday Apr 20, 2015 at 17:56 GMT

Note: approximate histograms can be computed using the histogram built-in (which computes a histogram off a reservoir sample).

@wesm wesm mentioned this issue Apr 21, 2015
Closed
@wesm wesm added the analytics Higher-level data analysis API related issues or PRs label Apr 29, 2015
@wesm wesm mentioned this issue May 28, 2015
Closed
wesm added a commit that referenced this issue May 31, 2015
@wesm
Implement bucket and histogram numeric expr transforms, category synt…
3d4d2b4 
…hetic type

Per #33 and #34

Author: Wes McKinney <wes@cloudera.com>

Closes #242 from wesm/bucket-transform and squashes the following commits:

ee037d6 [Wes McKinney] Slighly better label docs and move to analytics module
abb8797 [Wes McKinney] Add error checking for number of buckets
2bf3b51 [Wes McKinney] Implement label method for CategoryValue
c3609b3 [Wes McKinney] Casting bucket category to int32 is a noop
9b3b971 [Wes McKinney] Handle bucket edge cases and no-bucket under/over case
3abdb9b [Wes McKinney] Fix list repr interactive mode bug and tweak histogram base to avoid some FP error issues'
8dac292 [Wes McKinney] Initial histogram implementation, but interactive mode repr problems
dda0475 [Wes McKinney] Fix category type repr
f0404e3 [Wes McKinney] More exhaustive bucket test cases, and move dimension creation to translate_expr code path
cb90310 [Wes McKinney] Preliminary bucket implementation
013a5b9 [Wes McKinney] Implement basic category type and bucket and histogram APIs
@wesm wesm closed this as completed May 31, 2015
cpcloud pushed a commit that referenced this issue Jun 18, 2018
@xmnlab @cpcloud
[WiP] Adding new backend for MapD
037db67 
Also resolves #1418  and resolves #893

Author: Ivan Ogasawara <ivan.ogasawara@gmail.com>
Author: Ivan <ivan.ogasawara@gmail.com>
Author: Saul Shanabrook <s.shanabrook@gmail.com>

Closes #1419 from xmnlab/master and squashes the following commits:

f4a5dc8 [Ivan] Merge pull request #64 from xmnlab/master
8f5355f [Ivan Ogasawara] Removed xfail mark for ns unit
7043581 [Ivan] Merge pull request #63 from xmnlab/master
f9f017d [Ivan Ogasawara] Changed parameters for mapd tests.
5a342c8 [Ivan Ogasawara] Fixed small bugs; added test for ilike operation
d1e76b0 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
088f49d [Ivan] Merge pull request #62 from xmnlab/master
fe01689 [Ivan Ogasawara] Merged from master
825d5ca [Ivan] Merge pull request #61 from xmnlab/master
5fec8d4 [Ivan Ogasawara] Changed left and right params to column numeric type
a169dff [Ivan] Merge pull request #60 from xmnlab/master
2cac525 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
9b6a48c [Ivan Ogasawara] Corrections from the revision
cca6868 [Ivan] Merge pull request #59 from xmnlab/master
4f38f5a [Ivan Ogasawara] Skip tests for mapd about time units issues.
501d4cc [Ivan] Merge pull request #58 from xmnlab/master
0c2cc75 [Ivan Ogasawara] Changed requirements for CI python35.
0f423b8 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
c3dfc8b [Ivan] Merge pull request #57 from xmnlab/master
99b17ec [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
e30aca7 [Ivan] Merge pull request #56 from xmnlab/master
488808e [Ivan Ogasawara] 1st attempt to fix rewrite problem
3912ed6 [Ivan] Merge pull request #55 from xmnlab/master
abb659a [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
55a46fe [Ivan] Merge pull request #54 from xmnlab/master
8de95f8 [Ivan Ogasawara] Removed pin version from pandas
3983f40 [Ivan Ogasawara] Merged from master
7644a3d [Ivan] Merge pull request #53 from xmnlab/master
c224f15 [Ivan Ogasawara] Removed test_load_data from mapd.
20c8d88 [Ivan] Merge pull request #52 from xmnlab/master
58ec950 [Ivan Ogasawara] Changed mapd parameter order
ecbdb5a [Ivan] Merge pull request #51 from xmnlab/master
5927f25 [Ivan Ogasawara] Just add mapd backend to test when python == 3.
60401eb [Ivan] Merge pull request #50 from xmnlab/master
293fe94 [Ivan Ogasawara] Removed mapd from ci/waiter
505ea1a [Ivan] Merge pull request #49 from xmnlab/master
958f580 [Ivan Ogasawara] Change pandas to <0.23 on CI
52cd471 [Ivan Ogasawara] Change pandas to <0.23 on CI
994de1c [Ivan] Merge pull request #48 from xmnlab/master
8c2c826 [Ivan Ogasawara] Check on datamgr mapd and skip if python < 3
efcbc6a [Ivan] Merge pull request #47 from xmnlab/master
4c1d270 [Ivan Ogasawara] Fixed backend commented.
0d8284e [Ivan] Merge pull request #46 from xmnlab/master
89f17e9 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
25b60b8 [Ivan] Merge pull request #45 from xmnlab/master
49dc84c [Ivan Ogasawara] Skip tests for tableless exprs or python2.
db64334 [Ivan] Merge pull request #44 from xmnlab/master
f691a83 [Ivan Ogasawara] change load_data args
71d64e4 [Ivan Ogasawara] Added new args to load_data and added new test about load_data
6702de2 [Ivan] Merge pull request #43 from xmnlab/master
b73c78a [Ivan Ogasawara] Changed from load_table_arrow to load_table
0d1d814 [Ivan] Merge pull request #42 from xmnlab/master
f748136 [Ivan Ogasawara] Uncommented backend list.
57775b7 [Ivan] Merge pull request #41 from xmnlab/master
f19e4ee [Ivan Ogasawara] Removed qualified names to avoid MapD errors.
d991c66 [Ivan Ogasawara] Merged from upstream.
fd2bd73 [Ivan] Merge pull request #40 from xmnlab/master
c882e79 [Ivan Ogasawara] Rollback python27 env configuration.
a32a074 [Ivan] Merge pull request #39 from xmnlab/master
8656128 [Ivan Ogasawara] Added mapd.conf to allow some string operations.
1e5c0e7 [Ivan Ogasawara] Merge remote-tracking branch 'quansight/master'
63d91e7 [Saul Shanabrook] Fix database name in readme
a5138a0 [Ivan Ogasawara] *_FOAT operations removed.
71110e6 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
82b9d6b [Ivan Ogasawara] Changed some small issues
3ec3ea6 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
8fdcfc5 [Ivan] Merge pull request #38 from xmnlab/master
5a7aa1f [Ivan Ogasawara] Added support to thrift<=0.10.0
2c24c58 [Ivan] Merge pull request #37 from xmnlab/master
73c06a7 [Ivan Ogasawara] Added partial ref
c789d75 [Ivan] Merge pull request #36 from xmnlab/master
5c3522a [Ivan Ogasawara] Merged from master
e3d43c5 [Ivan] Merge pull request #35 from xmnlab/master
6f8b8b7 [Ivan Ogasawara] Fixed test
2455ed9 [Ivan] Merge pull request #34 from xmnlab/master
283c0cd [Ivan Ogasawara] Changed ApproxCountDistinc to HLLCardinality; Added IsNull operation;
b44ca33 [Ivan] Merge pull request #33 from xmnlab/master
53d4414 [Ivan Ogasawara] Added mapd to backend tests
465fafd [Ivan Ogasawara] Added mapd to tests
71c9366 [Ivan Ogasawara] Refactores appointed on PR
32321c2 [Ivan Ogasawara] Improved BinaryOp and Comparisson operation
dd2c15a [Ivan Ogasawara] Added docstring to distance function.
bc7fe8b [Ivan Ogasawara] Refactoring rad2deg and deg2rad
95e3f8d [Ivan Ogasawara] Improve datamgr mapd
5ba29e8 [Ivan] Merge pull request #32 from xmnlab/master
ca95daa [Ivan Ogasawara] Refactered.
a44f4bf [Ivan Ogasawara] [wip] refactoring
4b6a39e [Ivan] Merge pull request #31 from xmnlab/master
5559fd8 [Ivan Ogasawara] Changed circleci and appveyor confs.
20f3c75 [Ivan] Merge pull request #30 from xmnlab/master
14f230f [Ivan Ogasawara] MapD backend ibis_testing database creation.
ad5b9ae [Ivan] Merge pull request #29 from xmnlab/master
69d16c6 [Ivan Ogasawara] Changed CI conf
a8b5c7d [Ivan] Merge pull request #28 from xmnlab/master
4874872 [Ivan Ogasawara] Changed mapd_host.
bc11283 [Ivan Ogasawara] Using mapd docker
a5f95b5 [Ivan] Merge pull request #27 from xmnlab/master
07e88b5 [Ivan Ogasawara] Change default test host
e2b3c6a [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
066d606 [Ivan Ogasawara] CI using mapd server
c843d9c [Ivan] Merge pull request #26 from xmnlab/master
c906d1a [Ivan Ogasawara] Added context manager to MapD Client
d6bde23 [Ivan] Merge pull request #25 from xmnlab/master
e810730 [Ivan Ogasawara] Style code fixed.
02737c9 [Ivan Ogasawara] Added DDL and data storage operations.
90a41f1 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
31e69e6 [Ivan Ogasawara] Added create and drop database; create, alter and drop user;
1c30a89 [Ivan Ogasawara] Added create and drop database;
0608bb7 [Ivan Ogasawara] Adding insert from pandas dataframe.
ae92074 [Ivan Ogasawara] Adding DDL operations.
66fef38 [Ivan] Merge pull request #24 from xmnlab/master
8b0008e [Ivan Ogasawara] Increased CI waiter timeout
e664cfe [Ivan] Merge pull request #23 from xmnlab/master
4201127 [Ivan Ogasawara] Increase timeout CI waiter
bdcc7f2 [Ivan] Merge pull request #22 from xmnlab/master
b34f0c2 [Ivan Ogasawara] Code style bugs fixed
b1d0d5c [Ivan] Merge pull request #21 from xmnlab/master
834dcfc [Ivan Ogasawara] Added ci; docs; tests;
204a1e9 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
14062f2 [Ivan Ogasawara] 1st refectoring.
383ddd7 [Ivan Ogasawara] Resolves #1431 (ibis)
ca6e240 [Ivan Ogasawara] Removed date/timestamp diff. It should use diff between extracted date part.
228a8f5 [Ivan Ogasawara] Added timestamp/date diff; improve timestamp/date add/sub;
716494e [Ivan Ogasawara] Added byte_length operation
650c57b [Ivan Ogasawara] Added dateadd/timestampad mapd operations.
fa0719b [Ivan] Merge pull request #20 from xmnlab/master
0387ef7 [Ivan Ogasawara] Added count distinct; interval statement first version; added some direct extract data function;
d45db10 [Ivan] Merge pull request #18 from xmnlab/master
4d4d675 [Ivan Ogasawara] Fixed small text issue on README
366669d [Ivan] Merge pull request #17 from xmnlab/master
eea3b67 [Ivan Ogasawara] Fixed 90% of first comments PR bugs.
ab11e7b [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
c952e95 [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
70ba0ec [Ivan Ogasawara] Merge remote-tracking branch 'quansight/master'
bec6199 [Ivan Ogasawara] Added timestamp truncate and extract
0aac204 [Ivan Ogasawara] Added string operation; Added now and truncate operations;
703b997 [Ivan] Merge pull request #15 from xmnlab/master
c52a0bb [Ivan Ogasawara] Resolves ibis #1418 and resolves ibis #893
4061ea0 [Ivan Ogasawara] Added std, var and corr operations.
120efaa [Ivan Ogasawara] Merge remote-tracking branch 'ibis-project/master'
ed58893 [Ivan Ogasawara] Added small changes
3c4e6f7 [Ivan] Merge pull request #11 from xmnlab/master
bda8e3c [Ivan Ogasawara] Math, trigonometric and geometric functions added
5a06f30 [Ivan Ogasawara] Fixed small bugs.
14bcedf [Ivan] Merge pull request #9 from xmnlab/master
7b62e02 [Ivan Ogasawara] Changed TrigonometryUnary class
b0e0eb9 [Ivan Ogasawara] Sin function added.
f524894 [Ivan Ogasawara] Partially added mapd operations.
6592058 [Ivan Ogasawara] Resolves partially #5
a6b773a [Ivan Ogasawara] Double precision data type removed
33fc398 [Ivan Ogasawara] Resolves #4
77c93a5 [Ivan Ogasawara] Resolves #7
f34289d [Ivan Ogasawara] MapDCursor added
7289937 [Ivan] Merge pull request #8 from xmnlab/master
f381223 [Ivan Ogasawara] README updated; Initial changes to use execute method.
fb8bbe3 [Ivan] Merge pull request #3 from xmnlab/master
8c8df6d [Ivan Ogasawara] Improved mapd client and compiler; Added initial documentation.
6099ba2 [Ivan] Merge pull request #1 from xmnlab/master
b120529 [Ivan Ogasawara] MapDClient.table function worked (initial version).
f0d09f5 [Ivan Ogasawara] Changed connection
73becc5 [Ivan Ogasawara] Added mapd backend initial files.
amyskov pushed a commit to amyskov/ibis that referenced this issue Mar 31, 2020
@semelianova
Merge pull request ibis-project#33 from amyskov/santander
6570934 
[Santander]: ETL query implementation correction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
analytics Higher-level data analysis API related issues or PRs feature Features or general enhancements
Projects
None yet
Milestone
0.2
Development

No branches or pull requests
1 participant
@wesm