Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: BCP import memory leak with decimals #2455

Closed
1 task done
staticlibs opened this issue Apr 1, 2024 · 0 comments
Closed
1 task done

[Bug]: BCP import memory leak with decimals #2455

staticlibs opened this issue Apr 1, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@staticlibs
Copy link
Contributor

What happened?

There seems to be a leak in decimal (numeric) conversion. Memory is allocated in MessageContext, so it is released after BCP import finishes. Still can be a problem when importing large files.

Using the following table with a decimal column:

create table tab1 (col1 decimal(18,5))
insert into tab1 values ('41.42')
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
insert into tab1 select col1 from tab1
select count(*) from tab1
> 1048576

Export it with BCP:

bcp tab1 out tab1.bcp -n -S 192.168.122.1,1433 -U jdbc_user -P 12345678
...
1048576 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1453   Average : (721662.75 rows per sec.)

Clear the table:

truncate table tab1

Run server under mprof:

mprof run -M ./bin/postgres -D /home/alex/projects/postgres/dist/data

Import exported data back with BCP:

bcp tab1 in tab1.bcp -n -S 192.168.122.1,1433 -U jdbc_user -P 12345678
...
1048576 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 1578   Average : (664496.81 rows per sec.)

BCP import backend process shows excessive memory usage:

mprof plot

Figure_1

Will submit a patch to TdsTypeNumericToDatum shortly.

Version

BABEL_3_X_DEV (Default)

Extension

babelfishpg_tds

Which flavor of Linux are you using when you see the bug?

Fedora

Relevant log output

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct.
@staticlibs staticlibs added the bug Something isn't working label Apr 1, 2024
shardgupta pushed a commit that referenced this issue Apr 8, 2024
`TdsTypeNumericToDatum` [allocates a buffer](https://github.com/babelfish-for postgresql/babelfish_extensions/blob/6606b1118978477186869c3dddd2c6d85fcc6387/contrib/babelfishpg_tds/src/backend/tds/tdstypeio.c#L1133) for parsing incoming numeric data, but does not release this buffer. So it remains in `MessageContext` until the end of BCP import call.

Proposed patch frees this buffer before exiting `TdsTypeNumericToDatum`.

### Issues Resolved

#2455, BABEL-4882

Signed-off-by: Alex Kasko <alex@staticlibs.net>
shardgupta pushed a commit that referenced this issue Apr 8, 2024
`TdsTypeNumericToDatum` [allocates a buffer](https://github.com/babelfish-for postgresql/babelfish_extensions/blob/6606b1118978477186869c3dddd2c6d85fcc6387/contrib/babelfishpg_tds/src/backend/tds/tdstypeio.c#L1133) for parsing incoming numeric data, but does not release this buffer. So it remains in `MessageContext` until the end of BCP import call.

Proposed patch frees this buffer before exiting `TdsTypeNumericToDatum`.

### Issues Resolved

#2455, BABEL-4882

Signed-off-by: Alex Kasko <alex@staticlibs.net>
staticlibs added a commit to wiltondb/babelfish_extensions that referenced this issue Apr 14, 2024
…2456)

`TdsTypeNumericToDatum` [allocates a buffer](https://github.com/babelfish-for postgresql/babelfish_extensions/blob/6606b1118978477186869c3dddd2c6d85fcc6387/contrib/babelfishpg_tds/src/backend/tds/tdstypeio.c#L1133) for parsing incoming numeric data, but does not release this buffer. So it remains in `MessageContext` until the end of BCP import call.

Proposed patch frees this buffer before exiting `TdsTypeNumericToDatum`.

### Issues Resolved

babelfish-for-postgresql#2455, BABEL-4882

Signed-off-by: Alex Kasko <alex@staticlibs.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant