Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory Growth with PhraseListGrammar #1978

Closed
evanyounggenesys opened this issue Jun 13, 2023 · 8 comments
Closed

Memory Growth with PhraseListGrammar #1978

evanyounggenesys opened this issue Jun 13, 2023 · 8 comments
Assignees
@jhakulin
Labels
accepted Issue moved to product team backlog. Will be closed when addressed. bug Something isn't working

Comments

@evanyounggenesys
Copy link

Describe the bug
We've seen process RSS memory continually grow when using the speech-sdk for continuous recognition. I've been able to reproduce this when adding phrases to the PhraseListGrammar in a minimal reproduction below (updating the <subscription_key> value). From my testing, 1 phrase seems to be enough but the more phrases added the greater the growth. We're wondering if there is something wrong in this implementation to reclaim that memory or if there is an issue in the centos7 native libraries.

To Reproduce

public static void main(String[] args) throws InterruptedException, ExecutionException {
		System.loadLibrary("Microsoft.CognitiveServices.Speech.java.bindings");
		while(true) {
			// create resources
			SpeechConfig speechConfig = SpeechConfig.fromSubscription("<subscription_key>", "EastUS");
			speechConfig.setSpeechRecognitionLanguage("en-US");
			AudioStreamFormat audioStreamFormat = AudioStreamFormat.getWaveFormatPCM(8_000, (short) 16, (short) 1);
			PushAudioInputStream pushStream = AudioInputStream.createPushStream(audioStreamFormat);
			AudioConfig audioConfig = AudioConfig.fromStreamInput(pushStream);
			SpeechRecognizer speechRecognizer = new SpeechRecognizer(speechConfig, audioConfig);

			PhraseListGrammar phraseListGrammar = PhraseListGrammar.fromRecognizer(speechRecognizer);
			phraseListGrammar.addPhrase("Good morning");
			phraseListGrammar.addPhrase("Good evening");
			phraseListGrammar.addPhrase("Good afternoon");
			phraseListGrammar.addPhrase("Good night");
			phraseListGrammar.addPhrase("How are you");
			phraseListGrammar.addPhrase("How have you been");
			phraseListGrammar.addPhrase("How do you do");
			phraseListGrammar.addPhrase("How is it going");
			phraseListGrammar.addPhrase("It is nice to meet you");
			phraseListGrammar.addPhrase("Pleased to meet you");

			// start recognition
			speechRecognizer.startContinuousRecognitionAsync().get();

			// stop recognition
			speechRecognizer.stopContinuousRecognitionAsync().get();

			// close resources
			phraseListGrammar.close();
			speechRecognizer.close();
			audioConfig.close();
			pushStream.close();
			audioStreamFormat.close();
			speechConfig.close();
		}
	}

Version of the Cognitive Services Speech SDK
1.28.0

Platform, Operating System, and Programming Language

  • OS: Amazon Linux 2
  • Hardware - x86-64
  • Programming language: Java 11

Additional context
Environment variables used:
LD_LIBRARY_PATH=<path_to_speech_sdk_native>
SpeechSDK_UseCentos7Binaries=true
MALLOC_ARENA_MAX=2

JVM options:
-Xss256k -Xms15M -Xmx15M -XX:MaxDirectMemorySize=15M
@evanyounggenesys
Copy link
Author

I tried enabling the sdk logging with the property PropertyId.Speech_LogFilename. It didn't write to the file with the reproduction case here but worked when using a spring controller to run essentially the same code. In the log file, it looks like the phrases have TrackHandlers that I don't see a corresponding StopTracking call for and the total number keeps incrementing.

After 1 call to start, add 10 phrases, and stop:

[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43da460
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43da460, p=0x0x7f19a43da460, tot=1
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a444f9d0
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a444f9d0, p=0x0x7f19a444f9d0, tot=2
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a444fc40
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a444fc40, p=0x0x7f19a444fc40, tot=3
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a444ff60
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a444ff60, p=0x0x7f19a444ff60, tot=4
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43fefa0
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43fefa0, p=0x0x7f19a43fefa0, tot=5
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43ff1d0
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43ff1d0, p=0x0x7f19a43ff1d0, tot=6
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43ff4d0
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43ff4d0, p=0x0x7f19a43ff4d0, tot=7
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43ff590
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43ff590, p=0x0x7f19a43ff590, tot=8
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43ff780
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43ff780, p=0x0x7f19a43ff780, tot=9
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a43ff860
[173237]: 14353ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a43ff860, p=0x0x7f19a43ff860, tot=10

After 101 calls:

[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x55b50af0df30
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x55b50af0df30, p=0x0x55b50af0df30, tot=1001
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x55b50bd951e0
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x55b50bd951e0, p=0x0x55b50bd951e0, tot=1002
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a440f040
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a440f040, p=0x0x7f19a440f040, tot=1003
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x55b50bd70f60
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x55b50bd70f60, p=0x0x55b50bd70f60, tot=1004
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x55b50bd70e30
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x55b50bd70e30, p=0x0x55b50bd70e30, tot=1005
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a49eef60
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a49eef60, p=0x0x7f19a49eef60, tot=1006
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a47022c0
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a47022c0, p=0x0x7f19a47022c0, tot=1007
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a4702310
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a4702310, p=0x0x7f19a4702310, tot=1008
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a4322240
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a4322240, p=0x0x7f19a4322240, tot=1009
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  resource_manager.cpp:119 Created 'CSpxPhrase' as '236579156'
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:108 CSpxHandleTable::TrackHandle p=0x0x7f19a4702580
[654216]: 187233ms SPX_DBG_TRACE_VERBOSE:  handle_table.h:118 CSpxHandleTable::TrackHandle class=ISpxPhrase, h=0x0x7f19a4702580, p=0x0x7f19a4702580, tot=1010

The other TrackHandles appear to have a StopTracking called. The java sdk client only has addPhrase, clear, and close for PhraseListGrammar class. I tried clearing the PhraseListGrammar before closing it and still see this increase in memory and increase in total ISpxPhrase.

@BrianMouncer
Copy link
Contributor

@evanyounggenesys

That is strange. Are you only seeing this on CentOS7?
e.g. can you repro on windows or say Ubuntu 20.04 or CentOS8?
Do you see this memory growth if doing continuous recognition without loading phrase list grammars?

I am not a java expert, but your cleanup code above looks correct to me.
If you want to be extra pedantic about it, can you try doing the additional call to dispose() and set the object to null when done with it. That could in theory speed up garbage collection...

phraseListGrammar.close();
phraseListGrammar.dispose();
phraseListGrammar= null;

@BrianMouncer BrianMouncer self-assigned this Jun 14, 2023
@BrianMouncer BrianMouncer added bug Something isn't working accepted Issue moved to product team backlog. Will be closed when addressed. labels Jun 14, 2023
@evanyounggenesys
Copy link
Author

I'm able to reproduce this on a Mac (Big Sur) using the same reproduction code. I don't readily have a windows, ubuntu, or centOS8 machine to test on.

Removing all the phraseListGrammar.addPhrase and the memory stabilizes with this example.

The java 1.28.0 client-sdk has dispose as protected so I cannot call it. It looks like that is what close does though.

    public void close() {
        this.dispose(true);
    }

    protected void dispose(boolean var1) {
        if (!this.disposed) {
            super.dispose(var1);
            this.disposed = true;
        }
    }

I think this memory is allocated off heap as part of the client-sdk native functions and garbage collection will not reclaim that memory. I did try nulling the object with the reproduction example but I see the same memory growth.

@BrianMouncer
Copy link
Contributor

One of my teammates spotted what looks like a leak in the JNI layer.
I've opened internal bug #5319958 to track this.

Seems the same error is also in our C# binding (opened bug #5319969).

@evanyounggenesys
Copy link
Author

Awesome thanks. I don't have a clear reproduction case but I am seeing similar sdk logging for ISpxNamedProperties and ISpxRecognitionResult where the tot number seems to increase and not have the corresponding StopTracking. Could those be looked at for the JNI layer too? Not sure if those have a similar issue.

@evanyounggenesys
Copy link
Author

@BrianMouncer
After some more testing it looks like ISpxNamedProperties and ISpxRecognitionResult are eventually stopped in our use case. I haven't been able to reproduce any memory leak with those. Looking forward to the internal bug #5319958 fix for the phrases. Thanks!

@jhakulin jhakulin assigned jhakulin and unassigned BrianMouncer Jun 21, 2023
@jhakulin
Copy link

Internal workitem ref: 5319958

@jhakulin
Copy link

The fix will be released in the next release end of June.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jhakulin jhakulin

Labels
accepted Issue moved to product team backlog. Will be closed when addressed. bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jhakulin @evanyounggenesys @BrianMouncer