Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependency com.tdunning:t-digest to v3.3 #1402

Merged
merged 1 commit into from
Mar 2, 2023

Conversation

solrbot
Copy link
Collaborator

@solrbot solrbot commented Feb 26, 2023

This PR contains the following updates:

Package Type Update Change
com.tdunning:t-digest dependencies minor 3.1 -> 3.3

Release Notes

tdunning/t-digest

v3.2

===========
In release 3.2, the goal is to produce an update to the code given the large number of improvements since the previous release.

There are a few bugs that will survive this release, most notably in the AVLTreeDigest. These have to do with large numbers of repeated data points and are not new bugs.

There is also a lot of work going on with serialization. I need to hear from people about what they are doing with serialization so that we can build some test cases to allow an appropriate migration strategy to future serialization.

The paper continues to be updated. The algorithmic descriptions are getting reasonably clear, but the speed and accuracy sections need a complete revamp with current implementations.

Bugs, fixed and known

Fixed

The following important issues are fixed in this release

Issue #​90 Serialization for MergingDigest

Issue #​92 Serialization for AVLTreeDigest

Maybe fixed

This issue has substantial progress, but lacks a definitive test to determine whether it should be closed.

Issue 78 Stability under merging.

Pushed

The following issues are pushed beyond this release

Issue #​87 Future proof and extensible serialization

Issue #​89 Bad handling for duplicate values in AVLTreeDigest

All fixed issues

Here is a complete list of issues resolved in this release:

Issue #​55 Add time
decay to t-digest

Issue #​52 General
factory method for "fromBytes"

Issue #​90
Deserialization of MergingDigest BufferUnderflowException in 3.1

Issue #​92 Error in
AVLTreeDigest.fromBytes

Issue #​93 high
centroid frequency causes overflow - giving incorrect results

Issue #​67 Release of
version 3.2

Issue #​81
AVLTreeDigest with a lot of datas : integer overflow

Issue #​75 Adjusting
the centroid threshold values to obtain better accuracy at interesting
values

Issue #​74 underlying
distribution : powerlaw

Issue #​72 Inverse
quantile algorithm is non-contiguous

Issue #​65
totalDigest add spark dataframe column / array

Issue #​60 Getting
IllegalArgumentException when adding digests

Issue #​53
smallByteSize methods are very trappy in many classes -- should be
changed or have warnings in javadocs

Issue #​82 TDigest
class does not implement Serializable interface in last release.

Issue #​42 Histogram

Issue #​40 Improved
constraint on centroid sizes

Issue #​37 Allow
arbitrary scaling laws for centroid sizes

Issue #​29 Test
method testScaling() always adds values in ascending order

Issue #​84 Remove
deprecated kinds of t-digest

Issue #​76 Add
serializability

Issue #​77 Question:
Proof of bounds on merging digest size

Issue #​71 Simple
alternate algorithm using maxima, ranks and fixed cumulative weighting

Issue #​61 Possible
improvement to the speed of the algorithm

Issue #​58 jdk8
doclint incompatibility

Issue #​48 Build is
unstable under some circumstances

Issue #​63 Which
TDigest do you recommend?

Issue #​62 Very slow
performance; what am I missing?

Issue #​47 Make
TDigest serializable

Issue #​49
MergingDigest.centroids is wrong on an empty digest


Configuration

📅 Schedule: Branch creation - "* * * * *" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.


  • If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot

@solrbot solrbot force-pushed the renovate/com.tdunning-t-digest-3.x branch 2 times, most recently from 2761691 to d909ca2 Compare February 27, 2023 23:02
@solrbot solrbot force-pushed the renovate/com.tdunning-t-digest-3.x branch from d909ca2 to d8d369e Compare March 1, 2023 17:07
@risdenk risdenk merged commit b73b3fe into apache:main Mar 2, 2023
risdenk added a commit that referenced this pull request Mar 2, 2023
risdenk added a commit that referenced this pull request Mar 2, 2023
@risdenk
Copy link
Contributor

risdenk commented Mar 2, 2023

This caused these test failures so I reverted it:

ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testSubFacetStream (:solr:solrj-streaming)
    Test output: /Users/risdenk/repos/apache/solr/solr/solrj-streaming/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.io.stream.StreamExpressionTest.txt
    Reproduce with: gradlew :solr:solrj-streaming:test --tests "org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testSubFacetStream" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F0D3AF99EEE61E75 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testFacetStream (:solr:solrj-streaming)
    Test output: /Users/risdenk/repos/apache/solr/solr/solrj-streaming/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.io.stream.StreamExpressionTest.txt
    Reproduce with: gradlew :solr:solrj-streaming:test --tests "org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testFacetStream" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F0D3AF99EEE61E75 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testStatsStream (:solr:solrj-streaming)
    Test output: /Users/risdenk/repos/apache/solr/solr/solrj-streaming/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.io.stream.StreamExpressionTest.txt
    Reproduce with: gradlew :solr:solrj-streaming:test --tests "org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testStatsStream" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F0D3AF99EEE61E75 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.facet.DistributedFacetSimpleRefinementLongTailTest.test (:solr:core)
    Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.facet.DistributedFacetSimpleRefinementLongTailTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.facet.DistributedFacetSimpleRefinementLongTailTest.test" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F0D3AF99EEE61E75 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.facet.TestJsonFacets.testStatsDistrib {p0=SMART} (:solr:core)
    Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.facet.TestJsonFacets.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.facet.TestJsonFacets.testStatsDistrib {p0=SMART}" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F0D3AF99EEE61E75 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.facet.TestJsonFacets.testStats {p0=SMART} (:solr:core)
    Test output: /Users/risdenk/repos/apache/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.facet.TestJsonFacets.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.facet.TestJsonFacets.testStats {p0=SMART}" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F0D3AF99EEE61E75 -Ptests.file.encoding=ISO-8859-1

I don't know if I missed this in the check of the bot results above.

@solrbot solrbot deleted the renovate/com.tdunning-t-digest-3.x branch April 20, 2023 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants