Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case #2471

Merged
merged 2 commits into from
Mar 6, 2022
Merged

DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case #2471

merged 2 commits into from
Mar 6, 2022

Conversation

vdiravka
Copy link
Member

@vdiravka vdiravka commented Feb 22, 2022 •
edited

DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case

Description

Please check the description from DRILL-8145 and #2478

Documentation

N/A

Testing

This PR fully about tests

@vdiravka vdiravka force-pushed the DRILL-8145 branch 2 times, most recently from c8672f3 to eb6786e Compare February 27, 2022 16:50
@jnturton jnturton mentioned this pull request Mar 6, 2022
Closed
@jnturton
Copy link
Contributor

jnturton commented Mar 6, 2022

@vdiravka, looks like you've nailed it! Can I convert this PR out of draft and review it?

@vdiravka vdiravka changed the title Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case Mar 6, 2022
@vdiravka
DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled…
cc8a1af 
… test case

- changing timeout for TestDrillbitResilience tests
- timing tuning for memoryLeaksWhenCancelled
- update TestContainers version
- -DforkCount=1 for Travis Maven build
- directMemoryMb: 2500 -> 4500 leads to less occasinal test failures
@vdiravka
Copy link
Member Author

vdiravka commented Mar 6, 2022

@vdiravka, looks like you've nailed it! Can I convert this PR out of draft and review it?

Yes, I think it is ready. I didn't find optimal memory configs fo Travis build. Since there is 7.5Gb ENV memory limit. So there are 3 ways:

  1. long term: optimize Drill memory usage, like it was earlier
  2. to use paid plan for Travis.com builds
  3. to disable 2 forks for Maven build for Travis, because it uses more memory, than one fork. So it will be longer, but it will be less loaded on memory.

I've chosen third for now and found the build still faster than GitHub CI builds. So the solution looks fine to me.

@vdiravka vdiravka marked this pull request as ready for review March 6, 2022 15:58
@vdiravka vdiravka requested a review from jnturton March 6, 2022 15:58
jnturton
jnturton approved these changes Mar 6, 2022
View reviewed changes
Copy link
Contributor

@jnturton jnturton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, there is so much time consuming testing for this sort of fix.

@jnturton jnturton merged commit 7322e76 into apache:master Mar 6, 2022
@jnturton jnturton added the backport-to-stable This bug fix is applicable to the latest stable release and should be considered for inclusion there label May 10, 2022
jnturton added a commit to jnturton/drill that referenced this pull request May 11, 2022
@vdiravka @jnturton
DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled…
03a0dde 
… test case (apache#2471)

* Set Brotli codec jar and test to occur only on Linux amd64.

* DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case

- changing timeout for TestDrillbitResilience tests
- timing tuning for memoryLeaksWhenCancelled
- update TestContainers version
- -DforkCount=1 for Travis Maven build
- directMemoryMb: 2500 -> 4500 leads to less occasinal test failures

Co-authored-by: James Turton <james@somecomputer.xyz>
jnturton added a commit to jnturton/drill that referenced this pull request May 11, 2022
@vdiravka @jnturton
DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled…
be4e4a2 
… test case (apache#2471)

* Set Brotli codec jar and test to occur only on Linux amd64.

* DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case

- changing timeout for TestDrillbitResilience tests
- timing tuning for memoryLeaksWhenCancelled
- update TestContainers version
- -DforkCount=1 for Travis Maven build
- directMemoryMb: 2500 -> 4500 leads to less occasinal test failures

Co-authored-by: James Turton <james@somecomputer.xyz>
jnturton added a commit that referenced this pull request May 12, 2022
@jnturton @pseudomo
@rymarm @pjfanning @vvysotskyi @vdiravka @cgivre @luocooong @wojustme
DRILL-8221: Collect bug fixes from master for Drill 1.20.1 (#2545)
2fa849d 
* Prepare for the next bugfix iteration.

* SAS Reader fixes (#2472)

Co-authored-by: pseudomo <pseudomo@yandex.ru>

* Add jackson-bom (#2454)

* [DRILL-8150] log4j 2.17.2 in format-excel (#2476)

* DRILL-8151: Add support for more ElasticSearch and Cassandra data types (#2477)

* DRILL-8154: Upgrade to poi 5.2.1 (#2480)

* DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case (#2471)

* Set Brotli codec jar and test to occur only on Linux amd64.

* DRILL-8145: Fix flaky TestDrillbitResilience#memoryLeaksWhenCancelled test case

- changing timeout for TestDrillbitResilience tests
- timing tuning for memoryLeaksWhenCancelled
- update TestContainers version
- -DforkCount=1 for Travis Maven build
- directMemoryMb: 2500 -> 4500 leads to less occasinal test failures

Co-authored-by: James Turton <james@somecomputer.xyz>

* [MINOR UPDATE] Add Stalebot Config (#2487)

* [MINOR UPDATE] Fix license for Stalebot Config (#2488)

* DRILL-8156: Declare and chown a /data VOLUME in the Drill Dockerfile (#2491)

* Add a mountpoint and VOLUME for local storage to Dockerfile.

* Address review comments concerning layer ordering.

* Fix image size blowup by moving chmod to intermediate container.

* Combine RUN commands in Dockerfile.

* DRILL-8168: Do not duplicate attempts to impersonate a user in the REST API (#2495)

* DRILL-8172: Use the specified memory usage for Travis CI (#2500)

* DRILL-8165: Upgrade liquibase because of CVE-2022-0839 (#2497)

* Create codeql-analysis.yml

* Update codeql-analysis.yml

Removed cpp from code analysis

* [MINOR UPDATE] Add license to CodeQL YAML (#2501)

* DRILL-8176: Upgrade Jackson Due to CVE-2020-36518 (#2504)

* DRILL-8164: Upgrade metadata-extractor because of CVE-2022-24613 (#2493)

* DRILL-8164: Upgrade metadata-extractor because of CVE-2022-24613

* Update the ProfileCopyright tag name

* Update the mov format name

* Add the QuickTime.Rotation tag

* Bump metadata-extractor to 2.17.0

* DRILL-8178: Bump AWS Libraries to Latest Version (#2506)

* DRILL-8175: Update Drill release script after 1.20 (#2503)

* Set DRILL_PID_DIR in Dockerfile to writable location for distributed mode.

Some users of the images built from this Dockerfile customise
them so that they launch Drill in distributed mode instead of
embedded mode.  This change saves them from having to set
DRILL_PID_DIR themselves in order to succeed.

* Update release script and instructions after the release of 1.20.

- Add support for specifying a build profile such as "hadoop-2".
- Update instuctions for the Drill web site.
- Update instructions for uploading RCs (no more home.apache)
- Some fixes.

* DRILL-8176: minor issue in previous jackson bom (#2508)

* minor issue in previous jackson bom

* Update pom.xml

* DRILL-8187: Dialect factory returns ANSI SQL dialect for BigQuery (#2513)

* DRILL-8192: Cassandra queries fail when enabled Mongo plugin (#2518)

* DRILL-8013: Drill attempts to push "$SUM0" to JDBC storage plugin for AVG (#2521)

* DRILL-8194: Fix the function of REPLACE throws IndexOutOfBoundsException If text's length is more than previously applied (#2522)

* DRILL-8200: Update Hadoop libs to ≥ 3.2.3 for CVE-2022-26612 (#2525)

* Remove pointless Buffer casts.

Compiling Drill with JDK > 8 will still result in ByteBuffer <-> Buffer cast
exceptions at runtime when running on JDK 8 even though maven.target.version
is set to 8. Setting maven.compiler.release to 8 solves the Buffer casts
but raises a compilation error of package sun.security.jgss does not exist
for JDK 8. There were a few handwritten casts to avoid the Buffer casting
issue but many instances are not covered so the few reverted in this commit
achieve nothing.

* Update Hadoop to 3.2.3.

* [MINOR UPDATE] Update AWS Java SDK to 1.12.211

* DRILL-8219: Handle null catalog names returned by DB2 in storage-jdbc. (#2542)

Co-authored-by: pseudomo <yura_levchenko@mail.ru>
Co-authored-by: pseudomo <pseudomo@yandex.ru>
Co-authored-by: Rymar Maksym <rim.maxim+dev@gmail.com>
Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com>
Co-authored-by: Volodymyr Vysotskyi <vvovyk@gmail.com>
Co-authored-by: Vitalii Diravka <vitalii@apache.org>
Co-authored-by: Charles S. Givre <cgivre@apache.org>
Co-authored-by: luoc <luoc@apache.org>
Co-authored-by: xurenhe <xurenhe19910131@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jnturton jnturton jnturton approved these changes

Assignees
No one assigned
Labels
backport-to-stable This bug fix is applicable to the latest stable release and should be considered for inclusion there
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@vdiravka @jnturton