Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Every attach of fbclient/fbembed library to the host process leaks 64KB of memory [CORE1079] #1500

Closed
firebird-issue-importer opened this issue Jan 6, 2007 · 50 comments

Comments

@firebird-issue-importer
Copy link

firebird-issue-importer commented Jan 6, 2007

Submitted by: @dyemanov

Block progress on CORE1671

Votes: 1

Default pool is allocated during the module initialization but never freed. While this is not a problem for the dedicated server process, shared libraries should always cleanup properly.

The solution is to delete the default pool in the DLL_PROCESS_DETACH handler. I suspect the same issue may affect POSIX builds as well, but I'm clueless how it should be fixed there.

Commits: a57c2d0 39896cb dc1ea35

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 6, 2007

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 7, 2007

Commented by: @dyemanov

The fix was wrong, hence undone.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 7, 2007

Modified by: @dyemanov

status: Resolved [ 5 ] => Reopened [ 4 ]

resolution: Fixed [ 1 ] =>

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Apr 26, 2007

Modified by: @pcisar

Fix Version: 2.1 Beta 1 [ 10141 ]

Fix Version: 2.1.0 [ 10041 ] =>

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented May 15, 2007

Modified by: @dyemanov

Fix Version: 3.0.0 [ 10048 ]

Fix Version: 2.1 Beta 1 [ 10141 ] =>

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 9, 2008

Modified by: @AlexPeshkoff

assignee: Dmitry Yemanov [ dimitr ] => Alexander Peshkov [ alexpeshkoff ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 9, 2008

Modified by: @dyemanov

Target: 2.5.0 [ 10221 ]

Fix Version: 3.0.0 [ 10048 ] =>

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 17, 2008

Modified by: @AlexPeshkoff

priority: Minor [ 4 ] => Major [ 3 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 17, 2008

Modified by: @AlexPeshkoff

Link: This issue block progress on CORE1671 [ CORE1671 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 23, 2008

Commented by: @AlexPeshkoff

Leak was due to process memory pool not destroyed at library unload moment. Now it's destroyed exactly after running all destructors of global variables.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 23, 2008

Modified by: @AlexPeshkoff

status: Reopened [ 4 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5 Alpha 1 [ 10224 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 28, 2008

Modified by: @pcisar

Workflow: jira [ 11470 ] => Firebird [ 15395 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 28, 2008

Commented by: @ibprovider

Memory leaks still exists with 64-bit fbclient.dll on Windows Vista Ultimate x64

1000 load-unload has ~ 63MB leaks

In My test system I get ~ 4.5GB leaks :-)

For 32-bit fbclient.dll - no problems

Version of fbclient.dll : 2.5.0.21206

I'm used VS2005 SP1 for build x64 fbclient.dll

----
32-bit fbclient.dll has very small memory leaks for following operations: load-connect_db-disconect_db-unload

for 50000 passes I get ~5MB memory leaks

I'm tested the debug build of fbclient.dll (VS2005 SP1)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 28, 2008

Commented by: @asfernandes

I see the problem in 32-bit too. Your tests was with 700MB of virtual memory after some hours.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 29, 2008

Commented by: @ibprovider

I will to append to IBProvider the support for this "features" - rules for prevent unloading a client library. And I will execute the tests again :-)

Maybe IBP has memory leaks also. But the problems with load-unload (load-connect-disconnect-unload) I tested through direct work with fbclient.dll. At other system for tests.

And after unload a IBProvider, I see zero quantity of allocated blocks in my memory heaps

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 29, 2008

Commented by: @ibprovider

I have changed the IBProvider - added a mode for lock fbclient.dll in memory between tests [by default]

Now, the 32-bit and 64-bit tests work fine - no memory leaks :-)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 29, 2008

Commented by: @AlexPeshkoff

Confirm very small leak (looks like 24 bytes / per operation) on linux / x64. Have no ideas where it happens - but IMO not in our pools, they can't leak memory in such a small portions, at least PAGE_SIZE is leaked.

Adriano, what do you think - can it be INTL related? Can INTL still use sometimes plain new?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 29, 2008

Commented by: @asfernandes

> can it be INTL related?

In fbclient, certainly no.

Can INTL still use sometimes plain new?

Yes, for example in cs_icu.cpp: new BYTE[...

But for me (in Win32), Dmitry's tests with fbembed was unusable. After few minutes more than 1GB of memory was allocated and Windows become unstable.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 30, 2008

Commented by: @ibprovider

At current night I executed the long-play tests for BLOBs and ARRAYs. FB 2.5 (Super, x64, Windows Vista)

fbclient.dll was blocked in memory between tests (connections).

At end the test process (COM, IBProvider and fbclient.dll modules were unloaded) has been used 30.2MB memory.

Usualy, after "small" number of tests (~1 hour), test process is completed with 15-20 MB of memory.

I think, need run two pass of tests and compare with current results

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 30, 2008

Commented by: @AlexPeshkoff

Sorry - certainly it's not INTL bug - it happens in client too.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 30, 2008

Commented by: @AlexPeshkoff

Found a reason - bug in TLS support in NPTL (glibc 2.4). Replacing it with "old good" pthread_key_create() and family solves memory leak problem on linux.
I.e. mentioned leaks on windows are also OS specific.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Modified by: @dyemanov

status: Resolved [ 5 ] => Reopened [ 4 ]

resolution: Fixed [ 1 ] =>

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @AlexPeshkoff

Correction - leak was not due to glibc problem, but old gcc. Compiling with 4.3.1 solved a problem. I.e. it will not be present in our release.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @dyemanov

Just tried to load/unload fbclient.dll (64-bit, today's checkout) 1000 times. There was no memory growth noticed during the test run.

If I load/unload fbclient.dll from v2.1 by the same test program, I get about 64M of memory leak, as expected.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @asfernandes

Did you do at least one attach/detach in each load? It will be interest to know if leaks doesn't appear under this condition.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @ibprovider

I builded the new debug binaries of FB 2.5 x64 (current sources)

The simple load-unload - Yes, no memory leaks.

The "load-connect-disconnect-unload" - YES, no memory leaks, also.

It's miracle.

---
But I have got the crash for connect to database without parameters (empty database_path and empty DPB). But this is other history :-)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @dyemanov

I tried with: (a) no connections (just LoadLibrary/FreeLibrary), (b) same + attach/detach via XNET, (c) same + attach/detach via localhost.
No memory leaks on either the client or the server side.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @ibprovider

I tested the localhost connection only. No memory leaks.

XNET do not work on Vista x64 Ultimate (user account) (fbclient.dll x64)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Oct 31, 2008

Commented by: @dyemanov

Well, I suppose we can re-close this ticket then :-)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Nov 6, 2008

Commented by: @dyemanov

Re-closed as fixed, as nobody is able to prove this wrong :-)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Sep 21, 2009

Commented by: Robert Dollinger (rdollinger)

Exists a backport for this issue for 2.1.2 or 2.1.3?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Sep 21, 2009

Commented by: @AlexPeshkoff

Does not exist and sooner of all will not exist - at least until stable 2.5.1 release.
To fix an issue I had to rework all construction/destruction of global variables. And it's hard to enum a number of fixes we had to have to make it work on all supported platforms. If you have problems with client - please try to use 2.5 beta1, it appears not to have serious issues. What about embedded engine - sorry.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Sep 21, 2009

Commented by: Robert Dollinger (rdollinger)

Thanks for the speedy answer. Ok so I will test with the 2.5 Beta2.
Is there a way to go back from ODS 11.2 to ODS 11.1? Certainly not using new features of 2.5.
In other words can I restore a backup file made with FB 2.5 Beta2 tih FB 2.1.3?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Nov 12, 2009

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jun 11, 2010

Commented by: John Franck (bozzy)

Sorry to return to this topic but... I'm still seeing memory leaks when loading/unloading fbclient.dll (32bit).
I've tested these versions with these results (the environment is WinXP SP3):
(the values are per load/unload cycle)
2.1.3 - 12KB
2.5 Beta1 - 20KB
2.5 Beta2 - 20KB
2.5 RC1 - 16KB
2.5 RC2 - 16KB
So clearly something has changed, but from 2.1.3 things are getting worst...

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jun 11, 2010

Commented by: @dyemanov

Please try the latest snapshot build (marked as RC3).

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jun 12, 2010

Commented by: John Franck (bozzy)

Same thing, 16KB leak on fbclient.dll version 2.5.0.26038 (the most recent snapshot build I've found).

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jun 22, 2010

Commented by: John Franck (bozzy)

Dmitry (or someone else), any news?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Aug 10, 2010

Commented by: @AlexPeshkoff

Not reproduced on linux - no leaks after 1700 load/attach/detach/unload calls

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Aug 11, 2010

Commented by: @ibprovider

Windows build of fbclient.dll / 32bit / version 2.5.0.26054

Memory leaks on every load/unload still occurred

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Aug 11, 2010

Commented by: @dyemanov

A 16K leak is confirmed on Win64 as well.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Aug 11, 2010

Commented by: @dyemanov

Please test the next (tomorrow's) snapshot build, the leak should be fixed there.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Aug 11, 2010

Commented by: @ibprovider

Seems fbclient.dll works without memory leaks now :)

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Aug 11, 2010

Commented by: John Franck (bozzy)

Today's build 26054 still shows leaks, I'll test it tomorrow.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Nov 25, 2010

Commented by: @dyemanov

Commented by Robert Dollinger:

Hi, I have tested with Firebird 2.5.0.26074 final release (Classic Server) and there are still memory leaks on every load/unload on Windows Server 2008 64 bit.
On Windows 7 32 bit, seams there are no memory leaks.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Nov 25, 2010

Commented by: @dyemanov

Any more information? Do you just load/unload or attach a database as well? How big is the leak per every load/unload cycle? How did you measure it?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Nov 25, 2010

Commented by: @dyemanov

Commented by Robert Dollinger:

I made a little test-db as follow:

CREATE DATABASE '127.0.0.1/3250:E:\classictest\CLASSICTEST.FDB'
USER 'SYSDBA'
PAGE_SIZE 16384
DEFAULT CHARACTER SET UTF8 COLLATION UTF8;

CREATE TABLE NEW_TABLE (
"TRANSACTION_ID" BIGINT NOT NULL,
ZEITPUNKT TIMESTAMP,
CONNECTION BIGINT
);

ALTER TABLE NEW_TABLE ADD CONSTRAINT PK_NEW_TABLE PRIMARY KEY ("TRANSACTION_ID");

SET TERM ^ ;

CREATE OR ALTER TRIGGER NEW_TABLE_BI0 FOR NEW_TABLE
ACTIVE BEFORE INSERT POSITION 0
AS
begin
new."TRANSACTION_ID" = current_transaction;
new.connection = current_connection;
end
^
SET TERM ; ^

COMMIT WORK;

I my test programm then I make this 4 steps in a loop:

1. open the connection to the DB
2. insert into NEW_TABLE (Zeitpunkt) Values('now') returning transaction_id
3. select * from new_table where transaction_id=%i
4. And finally I close the connection.

Before I started my test, I have a physical memory usage of 60%. After ca. 5 minutes that the test-applications runs the usage increase to 70%.
When I stop my test-application the physical memory usage doesn't drop to 60% but to 65%.

I have tested running my test-application on the same machine as the fb_inet_server, and also on a different machine. Wiht the same result.

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Nov 25, 2010

Commented by: @dyemanov

The scenario above doesn't include loading/unloading the Firebird library. What did you mean priorly by load/unload?

Are you concerned about server or client memory usage? What physical memory counter do you monitor - global or per process? If the former, it's pointless, as it includes the things we cannot control, i.e. the OS file-system cache. If the latter, what process does occupy more and more memory while your test is running? Also, the Virtual Memory counter would be more useful than Physical Memory in this case. Finally, what database access layer is used in your application?

@firebird-issue-importer
Copy link
Author

firebird-issue-importer commented Jan 19, 2016

Modified by: @pavel-zotov

QA Status: No test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment