Skip to content

Memory leak when Archive Download fails #1235

Closed
#1240
Closed
Memory leak when Archive Download fails#1235
#1240
Labels
bug
Milestone
6.2.68

Description

@dpc22
dpc22
opened on Sep 9, 2021

Version

6.2.64

Installation method

My own rpm, derived from official source RPM. Redhat Enterprise Server 7.

Expected behavior

wwsympa.pl should not leak 4 GBytes of memory when an archive download fails.

Actual behavior

(Output from Linux top command):

16817 sympa     20   0 4916564   4.2g   6612 S   0.0 26.8   4:30.05 wwsympa.fcgi

4.2g is 4.2 Gytes of Residient Set Size. That starts as about 200 MBytes.

Additional Information

One of my list owners attempted to download a 1.3 GBytes ZIP archive of their mail archives (about 3 to 4 GBytes of raw data).

Apache timed out after (I think) 60 seconds: it is possible I need to increase the Apache default timeout. That left the wwsympa.fcgi process in question using 4.2 GBytes RSS, which does not appear to be returned to the system.

The list owner then repeated the process multiple times until all of the wwsympa processes were 4.2 GBytes in size.

I think that there might also be slow memory leaks in other Sympa daemons, but this largely explains why a virtual machine with 16 GBytes RAM was swapping like mad this morning until I restarted wwsympa. It normally only uses about 6 GBytes, with the rest spare/as buffer cache. OOM killer had stepped in, which is never a good sign on Linux.

I don't think that this memory leak happens if the download is successful. However I think that I will disable list archive downloads for the time being...

Activity

  1. ikedas
    added
    bug
    on Sep 10, 2021
  2. ikedas
    added this to the 6.2.66 milestone on Sep 10, 2021

  3. dpc22 commented on Sep 10, 2021

    @dpc22
    dpc22
    on Sep 10, 2021
    ContributorAuthor

    Possibly worth adding: the Linux "zip" command can build these archives much faster than the Perl modules Sympa is using.

  4. ikedas
    modified the milestones: 6.2.66, 6.2.68 on Sep 14, 2021
  5. ikedas
    added a commit that references this issue on Oct 1, 2021

    Memory consumption while archive download (sympa-community#1235).

    a42e0c6

  6. ikedas commented on Oct 3, 2021

    @ikedas
    ikedas
    on Oct 3, 2021
    Member

    @dpc22 , can you check this patch?
    This fix requires a new module Archive-Zip-SimpleZip.

  7. dpc22 commented on Oct 4, 2021

    @dpc22
    dpc22
    on Oct 4, 2021
    ContributorAuthor

    The patch applies, but Archive-Zip-SimpleZip isn't available from the RHEL base or EPEL repositories, so I will need to install it using CPAN or such like. All of the other Perl dependencies for Sympa are available from EPEL. I'm also a bit dubious about : "Module version: 0.040", although I guess version numbers are fairly arbitrary.

    wwsympa is creating a temporary .zip file in /var/spool/sympa/tmp. Is there a particular reason that we can't use /usr/bin/zip to generate that file if it is available? I would expect the command line tool to be much more readily available than Archive-Zip-SimpleZip.

  8. racke commented on Oct 4, 2021

    @racke
    racke
    on Oct 4, 2021
    Contributor

    For example, zip is not present on a basic Debian installation.

  9. dpc22 commented on Oct 4, 2021

    @dpc22
    dpc22
    on Oct 4, 2021
    ContributorAuthor

    Presumably you can install it using "apt get zip" though?

    I don't think that Archive-Zip-SimpleZip is available on Debian/Ubuntu either unless it is called something other than simplezip

  10. racke commented on Oct 4, 2021

    @racke
    racke
    on Oct 4, 2021
    Contributor

    Yes, you can install it of course. Just saying that we can't assume that zip is installed.

  11. ikedas commented on Oct 4, 2021

    @ikedas
    ikedas
    on Oct 4, 2021
    Member

    @dpc22, for the presnt, you have to install it manually, e.g. using cpanm (see the description).

    I think the problem is caused by the fact that Archive::Zip keeps the entire generated archive in memory. And Archive::Zip::SimpleZip seems to work around this problem.

    If Archive::Zip::SimpleZip would solve the problem, we will add it to dependency (see cpanfile file), and the packagers working for each distribution (Debian, RHEL/EPEL, FreeBSD, ...) will request thier community to include it into their distribution.

  12. dpc22 commented on Oct 4, 2021

    @dpc22
    dpc22
    on Oct 4, 2021
    ContributorAuthor

    I seem to have ended up in Perl dependency hell:

    lists-2[dpc22:Archive-Zip-SimpleZip-0.040]$ perl Makefile.PL 
Warning: prerequisite IO::Compress::Base 2.096 not found. We have 2.061.
Warning: prerequisite IO::Compress::Bzip2 2.096 not found. We have 2.061.
Warning: prerequisite IO::Uncompress::Base 2.096 not found. We have 2.061.

lists-2[dpc22:Archive-Zip-SimpleZip-0.021]$ perl Makefile.PL 
Warning: prerequisite IO::Compress::Base 2.081 not found. We have 2.061.
Writing Makefile for Archive::Zip::SimpleZip

    RHEL7 currently provides perl-IO-Compress-2.061
    RHEL8 currently provides perl-IO-Compress-2.081

    I will set up a throwaway VM to test the patch, but I don't want to fight the Redhat package manager on my production servers, which are carefully managed using Ansible. Archive-Zip-SimpleZip-0.021 looks plausible when we upgrade from RHEL 7 to RHEL 8 (probably next summer). In the mean time noone has actually complained that I have disabled archive downloads.

  13. dpc22 commented on Oct 4, 2021

    @dpc22
    dpc22
    on Oct 4, 2021
    ContributorAuthor

    Okay, I can confirm that the patch plus Archive-Zip-SimpleZip definitely fixes the issue that I reported. It takes about 25 seconds for Sympa to generate and a Web browser to download:

    -rw-rw-r-- 1 dpc22 dpc22 301M Oct  4 12:57  test-dpc22_archives.zip

    compared to about 20 seconds for the command line "zip" command to generate the raw archive, so not shabby at all. wwsympa.pl doesn't appear to have leaked substantial amounts of memory in the process. If I remove Archive::Zip::SimpleZip, the same archive download times out and leaks lots of memory just as before, but it has transparently fallen back to Archive::Zip.

    In order to install Archive::Zip::SimpleZip, cpanm needed to upgrade or install the following on RHEL7. Not as bad as I had feared, particularly given that cpanm seems to have installed its packages to /usr/local/share/perl5/ rather than /usr/share/perl5, which makes it simpler to revert.

    [root@lists-2 ~]# cpanm install Archive::Zip::SimpleZip
--> Working on install
Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/install-0.01.tar.gz ... OK
Configuring install-0.01 ... OK
Building and testing install-0.01 ... OK
Successfully installed install-0.01
--> Working on Archive::Zip::SimpleZip
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/Archive-Zip-SimpleZip-0.040.tar.gz ... OK
Configuring Archive-Zip-SimpleZip-0.040 ... OK
==> Found dependencies: IO::Uncompress::Base, IO::Compress::Base, IO::Compress::Bzip2
--> Working on IO::Uncompress::Base
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/IO-Compress-2.102.tar.gz ... OK
Configuring IO-Compress-2.102 ... OK
==> Found dependencies: Compress::Raw::Bzip2, Compress::Raw::Zlib
--> Working on Compress::Raw::Bzip2
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/Compress-Raw-Bzip2-2.101.tar.gz ... OK
Configuring Compress-Raw-Bzip2-2.101 ... OK
Building and testing Compress-Raw-Bzip2-2.101 ... OK
Successfully installed Compress-Raw-Bzip2-2.101 (upgraded from 2.061)
--> Working on Compress::Raw::Zlib
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/Compress-Raw-Zlib-2.101.tar.gz ... OK
Configuring Compress-Raw-Zlib-2.101 ... OK
Building and testing Compress-Raw-Zlib-2.101 ... OK
Successfully installed Compress-Raw-Zlib-2.101 (upgraded from 2.061)
Building and testing IO-Compress-2.102 ... OK
Successfully installed IO-Compress-2.102 (upgraded from 2.061)
Building and testing Archive-Zip-SimpleZip-0.040 ... OK
Successfully installed Archive-Zip-SimpleZip-0.040
5 distributions installed

  14. dpc22 commented on Oct 5, 2021

    @dpc22
    dpc22
    on Oct 5, 2021
    ContributorAuthor

    Now deployed on the live server. I will let you know if there turn out to be any complications. Thanks.

  15. ikedas commented on Oct 5, 2021

    @ikedas
    ikedas
    on Oct 5, 2021
    Member

    @dpc22, thanks for confirming!

    This PR will be merged for now, in some days. However if you notice anything, please let us know.

  16. ikedas
    closed this as completedin #1240on Oct 8, 2021
  17. ikedas
    added a commit that references this issue on Oct 8, 2021

    Merge pull request #1240 from ikedas/issue-1235 by ikedas

    63a5251

  18. iulistadmin commented on Dec 9, 2021

    @iulistadmin
    iulistadmin
    on Dec 9, 2021 · Hidden as off-topic

  19. ikedas commented on Dec 9, 2021

    @ikedas
    ikedas
    on Dec 9, 2021 · Hidden as off-topic

  20. iulistadmin commented on Dec 9, 2021

    @iulistadmin
    iulistadmin
    on Dec 9, 2021 · Hidden as off-topic
  21. ikedas
    added a commit that references this issue on Oct 18, 2024

    Giving up Archive-Zip

    062520a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions