-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-2675] Increase EVENT_QUEUE_CAPACITY by 20x. #1579
[SPARK-2675] Increase EVENT_QUEUE_CAPACITY by 20x. #1579
Conversation
Jenkins, test this please. |
QA tests have started for PR 1579. This patch merges cleanly. |
Is there an estimate on the size of an event ? How much memory would 200000 events take ? |
QA results for PR 1579: |
i want to know how much memory that @shivaram said before. @concretevitamin can show the number?thanks. |
Unfortunately I may not be able to find time to run an experiment soon. If anyone is interested, I think |
However many others also face this issue (evident from the mailing list). We should aim to fix this issue one way or another, either through this PR and/or the fix for SPARK-2316, by the 1.1 release. |
@concretevitamin would you mind closing this? The underlying issue has been fixed in #1679? |
This is another PR that removes `DisableBoson` from `JDBCV2Suite` as pie/boson#571 fixed the locale issue. Tested in my own rio https://rio.apple.com/projects/aci-kazuyuki-tanimura-boson/pipeline-specs/aci-kazuyuki-tanimura-boson-master-spark-sql-core-1-scan-only/pipelines/fb3190d6-ce12-4cca-b1c1-094a1e4d21fe https://rio.apple.com/projects/aci-kazuyuki-tanimura-boson/pipeline-specs/aci-kazuyuki-tanimura-boson-master-spark-sql-core-2-scan-only/pipelines/e873fb32-db32-4ab9-b126-7b536e24c7a8
JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2675
@pwendell @andrewor14