[Bug]: Memory leak in flair models

[ganga7445]

Describe the bug

I have fine-tuned the flair in my own ner dataset and using the below code. i have used xlmr transformer embeddings.
predict method is giving more memory leakage. @helpmefindaname
flair version is : 0.12.2

The

To Reproduce

from flair.models import SequenceTagger
import torch
tagger = SequenceTagger.load("best-model.pt")
from flair.data import Sentence

for text in [ "Financial Chair, Chi Phi Fraternity I Tuscaloosa, AL  Fall 2019",
              "Fort Smith Housekeeping Opportunities - Evenings, Data Scientist"]:
  sentence = Sentence(text)
  tagger.predict(sentence)
  entities = sentence.get_spans('ner')
  for entity in entities:
      print(f"Entity: {entity.text}, Label: {entity.tag}")
torch.cuda.empty_cache()

Expected behavior

Ideally, there should be no memory leaks. During testing with lower versions of Flair, no memory leaks were observed.

Logs and Stack traces

No response

Screenshots

No response

Additional Context

No response

Environment

Versions:

Flair

0.12.2

Pytorch

2.0.1+cu117

Transformers

4.30.2

GPU

True

[alanakbik]

Thanks for reporting this! @helpmefindaname can you check?

[helpmefindaname]

Hi @ganga7445 can you please upgrade to the latest version of flair and check if the issue persists?

[ganga7445]

@helpmefindaname @alanakbik
I tried with the latest versions of flair, torch, and transformers. still, the memory leak is happening.

[helpmefindaname]

Okay,
can you then elaborate more what you mean by memory leak & how you meassure it?

[ganga7445]

@helpmefindaname
Initially, loading the model demanded 1.5 gigabytes (GB) of RAM. Subsequently, after making predictions with the loaded model, there was a rise in RAM usage, ultimately resulting in a memory leak. I conducted debugging using the memory-profiler in Python, revealing that the predict method is allocating approximately 30 MB for each request.

[None-Such]

Is this a leak . . . or a configuration issue?

See - https://stackoverflow.com/questions/55322434/how-to-clear-cuda-memory-in-pytorch

[helpmefindaname]

@ganga7445
an I guessing right, that you are talking about https://pypi.org/project/memory-profiler/ ?
I tried that out, adopting your script to:

from memory_profiler import profile

from flair.models import SequenceTagger
import torch

tagger = SequenceTagger.load("ner")
from flair.data import Sentence


@profile
def main():
    for text in ["Financial Chair, Chi Phi Fraternity I Tuscaloosa, AL  Fall 2019",
                 "Fort Smith Housekeeping Opportunities - Evenings, Data Scientist"]:
        sentence = Sentence(text)
        tagger.predict(sentence)
        entities = sentence.get_spans('ner')
        for entity in entities:
            print(f"Entity: {entity.text}, Label: {entity.tag}")
    torch.cuda.empty_cache()


if __name__ == "__main__":
    main()

then running python -m memory_profiler scripts\memory_leak.py
gives output like this:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    10   3015.7 MiB   3015.7 MiB           1   @profile
    11                                         def main():
    12   3015.7 MiB      0.0 MiB           3       for text in ["Financial Chair, Chi Phi Fraternity I Tuscaloosa, AL  Fall 2019",
    13                                                          "Fort Smith Housekeeping Opportunities - Evenings, Data Scientist"]:
    14   3015.8 MiB   -214.3 MiB           2           sentence = Sentence(text)
    15   2801.8 MiB   -428.5 MiB           2           tagger.predict(sentence)
    16   2801.7 MiB     -0.0 MiB           2           entities = sentence.get_spans('ner')
    17   2801.7 MiB     -0.0 MiB           2           for entity in entities:
    18                                                     print(f"Entity: {entity.text}, Label: {entity.tag}")
    19   2801.7 MiB   -213.9 MiB           1       torch.cuda.empty_cache()

With the numbers on line 14 & 15 varying from run to run, but staying with negative increment.
I don't see any proof or indication of a memory leak here.

[ganga7445]

@helpmefindaname
You are right, but I developed a custom Named Entity Recognition (NER) model that uses different embeddings during training. However, I'm experiencing a memory leak issue during inference.

[helpmefindaname]

Okay, but as long as I cannot verify your claims and reproduce them, I cannot help you.
I'd suggest you try to find a reproducible example where that happens and share all the necessary details for it.

[danilyef]

I can confirm data leak as well, when I run model on pandas dataframe (text variable).
But for strange reason, if I sort dataframe by the length of the text variable (ascending order) memory leak issues disappears.

[helpmefindaname]

@danilyef
can you please create a reproducable example that demonstrates this leak and also shows how you determine that it is a memory leak?

[ganga7445]

@helpmefindaname
below is the code.

from flair.data import Sentence
from flair.models import SequenceTagger
from memory_profiler import profile

tagger = SequenceTagger.load('ner')
texts = [ "2014 Master of Computer Application (MCA) from Indira Gandhi Open",
"University(IGNOU).2010 Bachelor of Computer Application (BCA) from",
"Bangalore University",
"ITIL v3 Training conducted by Accenture - IT",
"TATA CONSULTANCY SERVICES : L2 Network Engineer -Jan 2017 till date",
"L2 & L2.5 Engineer for Morgan Stanley (BFSI Project):",
"Accenture : Network Engineer - Sept 2012 to 2017",
"Current Morgan Stanley, Bangalore",
"CISCO SWITCH -2960,Hp-procurve,etc..",
"Experience in managing networks for Trading sector network DATA Centers, Branch",
"CONVERGYS Nov 2010 - May 2012",
"MCA with 8 years and 3 months of experience in IT.",
"Completed CCNA (Cisco Certified Network Associate) with Cisco ID-",
"CSCO12765621",
"2014 Master of Computer Application (MCA) from Indira Gandhi Open",
"University(IGNOU).2010 Bachelor of Computer Application (BCA) from",
"Bangalore University",
"ITIL v3 Training conducted by Accenture - IT",
"TATA CONSULTANCY SERVICES : L2 Network Engineer -Jan 2017 till date",
"L2 & L2.5 Engineer for Morgan Stanley (BFSI Project):",
"Accenture : Network Engineer - Sept 2012 to 2017",
"Current Morgan Stanley, Bangalore",
"CISCO SWITCH -2960,Hp-procurve,etc..",
"Experience in managing networks for Trading sector network DATA Centers, Branch",
"CONVERGYS Nov 2010 - May 2012",
"MCA with 8 years and 3 months of experience in IT.",
"Completed CCNA (Cisco Certified Network Associate) with Cisco ID-",
"CSCO12765621",
"2014 Master of Computer Application (MCA) from Indira Gandhi Open",
"University(IGNOU).2010 Bachelor of Computer Application (BCA) from",
"Bangalore University",
"ITIL v3 Training conducted by Accenture - IT",
"TATA CONSULTANCY SERVICES : L2 Network Engineer -Jan 2017 till date",
"L2 & L2.5 Engineer for Morgan Stanley (BFSI Project):",
"Accenture : Network Engineer - Sept 2012 to 2017",
"Current Morgan Stanley, Bangalore",
"CISCO SWITCH -2960,Hp-procurve,etc..",
"Experience in managing networks for Trading sector network DATA Centers, Branch",
"CONVERGYS Nov 2010 - May 2012",
"MCA with 8 years and 3 months of experience in IT.",
"Completed CCNA (Cisco Certified Network Associate) with Cisco ID-",
"CSCO12765621",
"SandHI - A Science and Heritage Initiative, IIT Kharagpur | Prof. Joy Sen [May 2016 - June 2016]",
"Machine Learning | Python for Data Science | Probability and Statistics | Linear Algebra | Social Media Data Analytics | Programming and",
"Business Analyst | Merilytics [June 2019 - Dec 2019]",
"Credit Card Fraud Detection | Kaggle [June 2018]",
"Governor | Technology Adventure Society [May 2016 - April 2017]",
"One Two Flip Entertainment Pvt. Ltd. | Data Science [March 2018 - May 2018]",
"Business Development and Operations | ListUp [May 2017 - July 2017]",
"Multiword Fashion Term Extraction | Prof. Pawan Goyal [Aug 2018 - Oct 2018]",
"Programming Language/Libraries: Python, SQL, scikit-learn, Scala, pandas, beautiful-soup, Numpy, matplotlib",
"Softwares: Anaconda, Adobe Photoshop, Figma, Tableau, Microsoft Office - Advanced Excel, PowerPoint"]

@Profile
def prdict_test(texts):
metadata_sentences = [Sentence(sent) for sent in texts]
tagger.predict(metadata_sentences, mini_batch_size=32)
for sente in metadata_sentences:
ner_spans = sente.get_spans('ner')
entities = [{'text': span.text, 'label': span.tag,
'start_pos': span.start_position,
'end_pos': span.end_position,
'labels': {"value" : str(span.tag),
"confidence" : span.score}
}
for span in ner_spans]
model_result = {'text': sente.to_original_text(), 'entities': entities}

prdict_test(texts)

output is

using flair 0.13.1. does the issue with version of flair?

[ganga7445]

@alanakbik can you please check once?