Memory errors on Fedora 29 #516

qiray · 2019-12-06T12:35:03Z

Description

I have strange memory errors on Fedora 29 when use wordcloud package. Error messages are:

munmap_chunk(): invalid pointer

or

malloc(): invalid next size (unsorted)

or

Segmentation fault

Steps/Code to Reproduce

I run this code ('text' argument is plain text in Russian, 'filename' - something like 'output.png'):

Example:

# Bad example
from wordcloud import WordCloud

def make_wordcloud(text, filename):
    if len(text) == 0:
        print("Warning! No data for wordcloud")
        return
    wordcloud = WordCloud(background_color="#FFFFFF",
        width=800, height=400).generate(text)
    image = wordcloud.to_image()
    image.save(filename)

Something strange. When there are some unsused imports the code runs successfully.

# Good example
import matplotlib.pyplot as plt #unused import
from wordcloud import WordCloud
from gensim import corpora, models # and this one

def make_wordcloud(text, filename):
    if len(text) == 0:
        print("Warning! No data for wordcloud")
        return
    wordcloud = WordCloud(background_color="#FFFFFF",
        width=800, height=400).generate(text)
    image = wordcloud.to_image()
    image.save(filename)

Expected Results

I expect this code to run successfully.

Actual Results

I see "Segmentation fault" error. When run in GDB there is a backtrace like this one:

#0  0x00007ffff78e0e33 in free () from /lib64/libc.so.6
#1  0x00007fffd9962041 in raqm_layout () from /lib64/libraqm.so.0
#2  0x00007fffba5c373f in ?? () from /home/user/.local/lib/python3.7/site-packages/PIL/_imagingft.cpython-37m-x86_64-linux-gnu.so
#3  0x00007fffba5c3ab9 in ?? () from /home/user/.local/lib/python3.7/site-packages/PIL/_imagingft.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffff7d65444 in _PyMethodDef_RawFastCallKeywords () from /lib64/libpython3.7m.so.1.0
#5  0x00007ffff7d8040d in _PyMethodDescr_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#6  0x00007ffff7dcc0e0 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#7  0x00007ffff7d1daa8 in _PyEval_EvalCodeWithName () from /lib64/libpython3.7m.so.1.0
#8  0x00007ffff7d64cb1 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#9  0x00007ffff7dc7481 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#10 0x00007ffff7d1daa8 in _PyEval_EvalCodeWithName () from /lib64/libpython3.7m.so.1.0
#11 0x00007ffff7d64cb1 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#12 0x00007ffff7dc7481 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#13 0x00007ffff7d1daa8 in _PyEval_EvalCodeWithName () from /lib64/libpython3.7m.so.1.0
#14 0x00007ffff7d64cb1 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#15 0x00007ffff7dc8512 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#16 0x00007ffff7d1daa8 in _PyEval_EvalCodeWithName () from /lib64/libpython3.7m.so.1.0
#17 0x00007ffff7d64cb1 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#18 0x00007ffff7dc8512 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#19 0x00007ffff7d1daa8 in _PyEval_EvalCodeWithName () from /lib64/libpython3.7m.so.1.0
#20 0x00007ffff7d64cb1 in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#21 0x00007ffff7dc7481 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#22 0x00007ffff7d64b0a in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#23 0x00007ffff7dc7481 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#24 0x00007ffff7d64b0a in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#25 0x00007ffff7dc7481 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#26 0x00007ffff7d64b0a in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#27 0x00007ffff7dcbf79 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#28 0x00007ffff7d64b0a in _PyFunction_FastCallKeywords () from /lib64/libpython3.7m.so.1.0
#29 0x00007ffff7dc7639 in _PyEval_EvalFrameDefault () from /lib64/libpython3.7m.so.1.0
#30 0x00007ffff7d1daa8 in _PyEval_EvalCodeWithName () from /lib64/libpython3.7m.so.1.0
#31 0x00007ffff7d1e9d3 in PyEval_EvalCodeEx () from /lib64/libpython3.7m.so.1.0
#32 0x00007ffff7d1e9fb in PyEval_EvalCode () from /lib64/libpython3.7m.so.1.0
#33 0x00007ffff7e43732 in run_mod () from /lib64/libpython3.7m.so.1.0
#34 0x00007ffff7e43ac7 in PyRun_FileExFlags () from /lib64/libpython3.7m.so.1.0
#35 0x00007ffff7e464b8 in PyRun_SimpleFileExFlags () from /lib64/libpython3.7m.so.1.0
#36 0x00007ffff7e482a6 in pymain_main () from /lib64/libpython3.7m.so.1.0
#37 0x00007ffff7e484ec in _Py_UnixMain () from /lib64/libpython3.7m.so.1.0
#38 0x00007ffff787d413 in __libc_start_main () from /lib64/libc.so.6
#39 0x000055555555508e in _start ()

Versions

Linux-5.3.11-100.fc29.x86_64-x86_64-with-fedora-29-Twenty_Nine
Python 3.7.5 (default, Oct 17 2019, 12:21:00) 
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)]
NumPy 1.15.1
matplotlib 3.1.2
wordcoud 1.6.0

I have this error only in Fedora 29 (with different Python versions). When I run it in Linux Mint 19, Fedora 27 or Windows 10 everything seems well.

The text was updated successfully, but these errors were encountered:

amueller · 2019-12-06T19:11:12Z

That's really weird. How did you install the wordcloud package?

qiray · 2019-12-07T12:28:22Z

@amueller, as ususal way:

pip3 install wordcloud --user

amueller · 2019-12-11T22:07:31Z

Can you provide a self-contained example to reproduce? And what was the output of the pip install? Sent from phone. Please excuse spelling and brevity.

…

On Sat, Dec 7, 2019, 04:28 Yaroslav Zotov ***@***.***> wrote: @amueller <https://github.com/amueller>, as ususal way: pip3 install wordcloud --user — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#516?email_source=notifications&email_token=AADNYFVK643ZRP2KO7JP63DQXOJGPA5CNFSM4JWVZOL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGF3VQ#issuecomment-562847190>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADNYFQUI2LUKZA6RI6OX2TQXOJGPANCNFSM4JWVZOLQ> .

qiray · 2019-12-12T10:28:02Z

The complete example is here - https://gist.github.com/qiray/3c8bc3a78a29d46a76538f6fae6f4049

pip3 output:

Collecting wordcloud
  Using cached https://files.pythonhosted.org/packages/d6/af/55c7f2aa0997147943b474a74bab8deb17e7cf935b9abb8798d724c57721/wordcloud-1.6.0-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: matplotlib in /home/user/.local/lib/python3.7/site-packages (from wordcloud) (3.1.2)
Requirement already satisfied: pillow in /home/user/.local/lib/python3.7/site-packages (from wordcloud) (6.2.1)
Requirement already satisfied: numpy>=1.6.1 in /usr/lib64/python3.7/site-packages (from wordcloud) (1.15.1)
Requirement already satisfied: python-dateutil>=2.1 in /home/user/.local/lib/python3.7/site-packages (from matplotlib->wordcloud) (2.8.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/user/.local/lib/python3.7/site-packages (from matplotlib->wordcloud) (2.4.5)
Requirement already satisfied: cycler>=0.10 in /home/user/.local/lib/python3.7/site-packages (from matplotlib->wordcloud) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/user/.local/lib/python3.7/site-packages (from matplotlib->wordcloud) (1.1.0)
Requirement already satisfied: six>=1.5 in /home/user/.local/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib->wordcloud) (1.13.0)
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->wordcloud) (40.8.0)
Installing collected packages: wordcloud
Successfully installed wordcloud-1.6.0

amueller · 2019-12-23T20:15:06Z

Sorry for the slow reply. Can you try building wordcloud from scratch on your system and see if you observe the same issue?

amueller · 2019-12-23T20:16:14Z

though it seems an issue in PIL, not in wordcloud... Maybe reporting to PIL or fedora would make sense?

qiray · 2020-01-13T10:31:52Z

It looks like rebuilding wordcloud doesn't help. I think I'll update Fedora later and check it again.

amueller · 2020-01-13T19:51:19Z

Can you see if you can reproduce with just PIL?

qiray · 2020-01-29T09:15:13Z

Can you see if you can reproduce with just PIL?

I tried some PIL examples and it seems to be correct.

amueller · 2020-02-01T23:27:48Z

Can you open an issue there please?

qiray · 2020-02-04T08:46:14Z

Well, I can but I can't reproduce this error using PIL only.

amueller · 2020-02-04T14:56:17Z

Oh sorry I misread your reply. Can you isolate the call in wordcloud that produces the failure then? It looks like it is a PIL call.

qiray · 2020-02-11T14:08:29Z

I found that it crashes in file wordcloud/wordcloud.py in method generate_from_frequencies() at line:

box_size = draw.textsize(word, font=transposed_font)

I also tried to make a wordcloud with a custom font but had same result.

qiray · 2020-02-13T18:00:19Z

Some updates.

I tried clean install of new Fedora 31 and it seems to be correct. So it maybe was an error in my distributive.

shirzady1934 · 2020-04-20T19:04:13Z

I had the same issue in fedora 30 and solved it by downgrading pillow version to 6.2.2.
pip install --no-binary=Pillow Pillow==6.2.2 --force

mbollmann · 2020-05-23T13:18:01Z

Same thing just happened to me in Fedora 31, wordcloud==1.70 and Pillow=7.1.2. Downgrading Pillow to 6.2.2 helped, didn't try anything else.

winng51 · 2020-11-23T06:34:03Z

met the same problem, and successfully got the picture after import the packages...(using Pillow 8.0.1)

import matplotlib.pyplot as plt
from gensim import corpora, models

finally I install the Pillow 6.2.2 and running good

shirzady1934 · 2020-11-24T16:10:29Z

Some update for this issue:
this problem will be resolve for installing any version of Pillow by compiling from the source and passing a specific install option, by running this command.
pip install -U --compile --install-option=-O1 Pillow --user

joefromct · 2020-12-09T16:48:08Z

FYI, @shirzady1934 's fix did it for me, but I did have to pass in --ignore-installed as I did this fix after things were already there.

I also didn't use --user as I'm in a venv.

Full command:

 pip install --ignore-installed -U --compile --install-option=-O1 Pillow

dlukes · 2021-01-28T20:16:35Z

AFAIK this is being worked on upstream, the root cause has hopefully been identified in python-pillow/Pillow#4225 and there's a PR that looks promising in python-pillow/Pillow#5062. So this issue can probably be closed?

radarhere · 2021-04-02T21:41:02Z

The PR has been merged and released as part of Pillow 8.2.0

amueller mentioned this issue Feb 26, 2021

Unexplainable errors when generating SVG #626

Open

amueller closed this as completed Feb 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory errors on Fedora 29 #516

Memory errors on Fedora 29 #516

qiray commented Dec 6, 2019

amueller commented Dec 6, 2019

qiray commented Dec 7, 2019

amueller commented Dec 11, 2019 via email

qiray commented Dec 12, 2019

amueller commented Dec 23, 2019

amueller commented Dec 23, 2019

qiray commented Jan 13, 2020

amueller commented Jan 13, 2020

qiray commented Jan 29, 2020

amueller commented Feb 1, 2020

qiray commented Feb 4, 2020

amueller commented Feb 4, 2020

qiray commented Feb 11, 2020 •

edited

Loading

qiray commented Feb 13, 2020

shirzady1934 commented Apr 20, 2020 •

edited

Loading

mbollmann commented May 23, 2020

winng51 commented Nov 23, 2020

shirzady1934 commented Nov 24, 2020

joefromct commented Dec 9, 2020 •

edited

Loading

dlukes commented Jan 28, 2021

radarhere commented Apr 2, 2021

Memory errors on Fedora 29 #516

Memory errors on Fedora 29 #516

Comments

qiray commented Dec 6, 2019

Description

Steps/Code to Reproduce

Expected Results

Actual Results

Versions

amueller commented Dec 6, 2019

qiray commented Dec 7, 2019

amueller commented Dec 11, 2019 via email

qiray commented Dec 12, 2019

amueller commented Dec 23, 2019

amueller commented Dec 23, 2019

qiray commented Jan 13, 2020

amueller commented Jan 13, 2020

qiray commented Jan 29, 2020

amueller commented Feb 1, 2020

qiray commented Feb 4, 2020

amueller commented Feb 4, 2020

qiray commented Feb 11, 2020 • edited Loading

qiray commented Feb 13, 2020

shirzady1934 commented Apr 20, 2020 • edited Loading

mbollmann commented May 23, 2020

winng51 commented Nov 23, 2020

shirzady1934 commented Nov 24, 2020

joefromct commented Dec 9, 2020 • edited Loading

dlukes commented Jan 28, 2021

radarhere commented Apr 2, 2021

qiray commented Feb 11, 2020 •

edited

Loading

shirzady1934 commented Apr 20, 2020 •

edited

Loading

joefromct commented Dec 9, 2020 •

edited

Loading