Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

42.2.5 poor performance with bulk copy #1312

Closed
1 of 2 tasks
niels1voo opened this issue Oct 18, 2018 · 4 comments
Closed
1 of 2 tasks

42.2.5 poor performance with bulk copy #1312

niels1voo opened this issue Oct 18, 2018 · 4 comments

Comments

@niels1voo
Copy link

@niels1voo niels1voo commented Oct 18, 2018

I'm submitting a ...

  • bug report
  • feature request

Describe the issue
Copy performance is very poor under driver version 42.2.5. We tested several versions prior to 42.2.5 and found them equivalent in performance. 42.2.5 was several orders of magnitude worse.

Java Version
1.8.0_181
OS Version
MacOS 10.14
PostgreSQL Version
9.6.10
To Reproduce
Steps to reproduce the behaviour:
(you may want to edit TestPedalDialect:205 to reduce the number of iterations, power 16 would suffice)
With version 42.2.4
clone https://github.com/niels1voo/pedal-dialect, modify JPAConfiguration lines 52-54 to suite your local db, run TestPedalDialect.testBulkCopyCommand.
notice the log info messages from CopyCommandImpl:90.
With 42.2.5
Edit pom.xml to change the postgresql dependency to version 42.2.5
Re-run TestPedalDialect.testBulkCopyCommand
Execution time slows by several orders of magnitude
Expected behaviour
42.2.5 should perform about as well as previous versions (500K inserts in 21 seconds)
And what actually happens
42.2.5 performs much, much worse than previous versions (500K inserts in 726 seconds)

Logs
pgjdbc-trace-42-2-4.log.gz
pgjdbc-trace-42-2-5.log.gz

@niels1voo niels1voo changed the title poor performanc with bulb copy poor performanc with bulk copy Oct 18, 2018
@niels1voo niels1voo changed the title poor performanc with bulk copy poor performance with bulk copy Oct 18, 2018
@niels1voo niels1voo changed the title poor performance with bulk copy 42.2.5 poor performance with bulk copy Oct 18, 2018
@davecramer
Copy link
Member

@davecramer davecramer commented Oct 18, 2018

is there a simple way to run this with mvn?

@davecramer
Copy link
Member

@davecramer davecramer commented Oct 18, 2018

running in intellij, gives me:
Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException

don't suppose you could make this simpler ?

@niels1voo
Copy link
Author

@niels1voo niels1voo commented Oct 18, 2018

mvn -Dtest=TestPedalDialect#testBulkCopyCommand test
but you still need to setup the db per your test db preference.

@davecramer
Copy link
Member

@davecramer davecramer commented Oct 18, 2018

OK, I can replicate this... thx

vlsi added a commit to vlsi/pgjdbc that referenced this issue Oct 18, 2018
PGStream#hasMessagePending should not perform network requests otherwise it causes per row delay in async copy

fixes pgjdbc#1312
vlsi added a commit to vlsi/pgjdbc that referenced this issue Oct 19, 2018
PGStream#hasMessagePending should not perform network requests otherwise it causes per row delay in async copy

fixes pgjdbc#1312
vlsi added a commit to vlsi/pgjdbc that referenced this issue Oct 21, 2018
PGStream#hasMessagePending should not perform network requests otherwise it causes per row delay in async copy

fixes pgjdbc#1312
vlsi added a commit to vlsi/pgjdbc that referenced this issue Oct 21, 2018
PGStream#hasMessagePending should not perform network requests otherwise it causes per row delay in async copy

fixes pgjdbc#1312
vlsi added a commit to vlsi/pgjdbc that referenced this issue Oct 24, 2018
PGStream#hasMessagePending should not perform network requests otherwise it causes per row delay in async copy

fixes pgjdbc#1312
@vlsi vlsi closed this in e2623d6 Oct 25, 2018
davecramer added a commit to davecramer/pgjdbc that referenced this issue Jul 5, 2021
The problem is SSL sockets use buffers for input, and `stream#available()` might easily return 0 even though the data is available in the downstream socket. The only viable WA is to perform `read` calls, however it might impact performance (e.g. for async copy operations). The resolution is to throttle read calls for copy api, and keep 1ms reads for replication api.

fixes pgjdbc#1312
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants