Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`conda env create` eats up memory #5003

Open
alanhdu opened this issue Apr 5, 2017 · 5 comments
Open

`conda env create` eats up memory #5003

alanhdu opened this issue Apr 5, 2017 · 5 comments

Comments

@alanhdu
Copy link
Contributor

@alanhdu alanhdu commented Apr 5, 2017

I have a DigitalOcean droplet with 512 MB of RAM (although ~300 MB are used by other processes on the server), and I'm consistently running out of RAM when trying to run conda env create.

On a fresh continuumio/miniconda docker image, /usr/bin/time -v conda env create --name test --file environment.yml uses ~280 MB of RAM! Running the same command twice still uses ~250 MB of RAM, so the memory isn't coming from the download stage.

Information:

$ conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.11
       conda is private : False
      conda-env version : 4.3.11
    conda-build version : not installed
         python version : 2.7.13.final.0
       requests version : 2.12.4
       root environment : /opt/conda  (writable)
    default environment : /opt/conda
       envs directories : /opt/conda/envs
                          /root/.conda/envs
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False
             user-agent : conda/4.3.11 requests/2.12.4 CPython/2.7.13 Linux/4.10.0-14-generic debian/8.5 glibc/2.19
                UID:GID : 0:0

The environment.yml file:

name: test
channels:
- conda-forge
dependencies:
- python=3.6

The RAM usage

$ /usr/bin/time -v conda env create --name test --file environment.yml 
Fetching package metadata ...........
Solving package specifications: .
#
# To activate this environment, use:
# > source activate test
#
# To deactivate this environment, use:
# > source deactivate test
#

        Command being timed: "conda env create --name test --file environment.yml"
        User time (seconds): 10.58
        System time (seconds): 0.86
        Percent of CPU this job got: 95%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:12.02
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 251912
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 104056
        Voluntary context switches: 37
        Involuntary context switches: 316
        Swaps: 0
        File system inputs: 16
        File system outputs: 39336
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
@alanhdu

This comment has been minimized.

Copy link
Contributor Author

@alanhdu alanhdu commented Apr 5, 2017

Interestingly, with the following environment.yml file:

name: test
dependencies:
- python=3.6

peak memory usage drops to ~140 MB:

        Command being timed: "conda env create --name test --file environment.yml"
        User time (seconds): 10.66
        System time (seconds): 1.65
        Percent of CPU this job got: 27%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:44.70
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 137592
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 104914
        Voluntary context switches: 12506
        Involuntary context switches: 1191
        Swaps: 0
        File system inputs: 200
        File system outputs: 202296
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

So it looks like including channels in the environment.yml boosts memory usage by >100 MB of RAM!

@alanhdu

This comment has been minimized.

Copy link
Contributor Author

@alanhdu alanhdu commented Apr 5, 2017

Update: running conda update --all --yes in the continuumio/miniconda doesn't seem to change anything:

$ conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.16
       conda is private : False
      conda-env version : 4.3.16
    conda-build version : not installed
         python version : 2.7.13.final.0
       requests version : 2.13.0
       root environment : /opt/conda  (writable)
    default environment : /opt/conda
       envs directories : /opt/conda/envs
                          /root/.conda/envs
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /root/.condarc
           offline mode : False
             user-agent : conda/4.3.16 requests/2.13.0 CPython/2.7.13 Linux/4.10.0-14-generic debian/8.5 glibc/2.19
                UID:GID : 0:0
$ /usr/bin/time -v conda env create                                                                                                                       
Fetching package metadata ...........
Solving package specifications: .
#
# To activate this environment, use:
# > source activate test
#
# To deactivate this environment, use:
# > source deactivate test
#

        Command being timed: "conda env create"
        User time (seconds): 5.42
        System time (seconds): 0.73
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.13
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 250688
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 98535
        Voluntary context switches: 33
        Involuntary context switches: 380
        Swaps: 0
        File system inputs: 0
        File system outputs: 39336
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
@alanhdu alanhdu changed the title Very High Memory Usage `conda env create` eats up memory Apr 5, 2017
@alanhdu

This comment has been minimized.

Copy link
Contributor Author

@alanhdu alanhdu commented Apr 5, 2017

Hm... the plot thickens. On my laptop (not the Docker image), conda env create only takes ~170 MB, with or without the

channels:
- conda-forge

as part of the environment.yml. Not sure what's going on here.

I'll take a look with tracemalloc, but getting more information might take a while (I've never used tracemalloc before, so this will be a learning experience 😺).

$ /usr/bin/time -v conda env create
Fetching package metadata .............
Solving package specifications: .
#
# To activate this environment, use:
# > source activate test
#
# To deactivate this environment, use:
# > source deactivate test
#

	Command being timed: "conda env create"
	User time (seconds): 4.06
	System time (seconds): 0.58
	Percent of CPU this job got: 63%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.27
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 169672
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 64928
	Voluntary context switches: 84
	Involuntary context switches: 75
	Swaps: 0
	File system inputs: 0
	File system outputs: 42872
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
@kalefranz

This comment has been minimized.

Copy link
Member

@kalefranz kalefranz commented Apr 5, 2017

@kalefranz

This comment has been minimized.

Copy link
Member

@kalefranz kalefranz commented May 10, 2017

There are a few tickets, some very old, complaining about memory issues. So nothing new FWIW. But it's something we are keeping in mind for the future.

dlj-NaN added a commit to protocolbuffers/protobuf that referenced this issue Mar 10, 2020
We can remove the dependency on sphinxcontrib, since sphinx.ext.napoleon is
already available. This means we can drop conda-forge, which substantially
reduces build memory usage (see: readthedocs/readthedocs.org#6742,
conda/conda#5003, conda/conda#8650, readthedocs/readthedocs.org#6692).

Bump Sphinx to 2.4.0, while we're at it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.