Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use App::cpm to install perlcritic faster, and clean .perl-cpm cache #3036

Merged
merged 4 commits into from Oct 23, 2023

Conversation

echoix
Copy link
Collaborator

@echoix echoix commented Oct 23, 2023

The installation of perlcritic left some cache files in the image, about 31 MB. This PR fixes that.
Also, I tried out other installation methods, and to see if I was able to remove the perl-dev dependency (I couldn't).
App::cpm is a fast CPAN module installer, and they claim it is 3x faster than cpanm. It can use multiple threads to parallelize the work to do. This PR uses that.

I used the tool called dive to look into the images, and hyperfine to benchmark building of the perl_perlcritic image.
Results are :

Command Mean [s] Min [s] Max [s] Relative Image size [MB] Space Saved [MB]
docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus . 78.554 ± 0.821 78.020 79.499 1.49 ± 0.04 380 -
docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus_clean" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus-clean . 75.322 ± 2.437 73.781 78.131 1.43 ± 0.06 349 31
docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm . 56.371 ± 2.916 53.776 59.526 1.07 ± 0.06 346 34.2
docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe . 52.754 ± 1.246 51.494 53.986 1.00 345 35.4
docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe_workers2" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe-workers-2 . 58.401 ± 0.637 57.833 59.090 1.11 ± 0.03 345 35.4
Details of run time

hyperfine --min-runs 3  --export-markdown timings_perlcritic5.md  --export-json timings_perlcritic5.json  \
    'docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus . ' \
    'docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus_clean" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus-clean . ' \
    'docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm . '  \
    'docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe . ' \
    'docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe_workers2" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe-workers-2 . '
Benchmark 1: docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus . 
  Time (mean ± σ):     78.554 s ±  0.821 s    [User: 0.203 s, System: 0.198 s]
  Range (min … max):   78.020 s … 79.499 s    3 runs
 
Benchmark 2: docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus_clean" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus-clean . 
  Time (mean ± σ):     75.322 s ±  2.437 s    [User: 0.223 s, System: 0.169 s]
  Range (min … max):   73.781 s … 78.131 s    3 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 3: docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm . 
  Time (mean ± σ):     56.371 s ±  2.916 s    [User: 0.201 s, System: 0.180 s]
  Range (min … max):   53.776 s … 59.526 s    3 runs
 
Benchmark 4: docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe . 
  Time (mean ± σ):     52.754 s ±  1.246 s    [User: 0.174 s, System: 0.187 s]
  Range (min … max):   51.494 s … 53.986 s    3 runs
 
Benchmark 5: docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe_workers2" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe-workers-2 . 
  Time (mean ± σ):     58.401 s ±  0.637 s    [User: 0.208 s, System: 0.160 s]
  Range (min … max):   57.833 s … 59.090 s    3 runs
 
Summary
  docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe .  ran
    1.07 ± 0.06 times faster than docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm . 
    1.11 ± 0.03 times faster than docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpm_pipe_workers2" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpm-pipe-workers-2 . 
    1.43 ± 0.06 times faster than docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus_clean" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus-clean . 
    1.49 ± 0.04 times faster than docker buildx build --pull --rm -f "linters/perl_perlcritic/Dockerfile_cpanminus" --progress plain --no-cache -t megalinter-only-perl_perlcritic:test-echoix-cpanminus . 

Details of disk usage

Using dive with a command similar to:

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive:latest megalinter-only-perl_perlcritic:test-echoix-cpm-pipe

Note that there is missing the -g global flag in the installation screenshots for cpm, but the sizes are the same, only installed in another folder to be found without changing any paths.

Original installation:
image

Original installation, cleaning /root/.cpanm folder:
image

Installing and using App::Cpm, cleaning the /root/.perl-cpm folder:
image

Using cpm directly from a pipe, without installing it:
image

Using cpm directly from a pipe, without installing it, and limiting to 2 workers, the closest I can get to what it will be on GitHub Actions runners:
image

I also tested linting a Perl repo locally with the new installation method, and works as the original

Proposed Changes

  1. Use cpm directly from pipe to install Perl Critic
  2. Clean perl cache in same step

Readiness Checklist

Author/Contributor

  • Add entry to the CHANGELOG listing the change and linking to the corresponding issue (if appropriate)
  • If documentation is needed for this change, has that been included in this pull request

Reviewing Maintainer

  • Label as breaking if this is a large fundamental change
  • Label as either automation, bug, documentation, enhancement, infrastructure, or performance

@echoix echoix added dependencies Pull requests that update a dependency file docker Pull requests that update Docker code performance labels Oct 23, 2023
@echoix echoix temporarily deployed to dev October 23, 2023 04:33 — with GitHub Actions Inactive
@echoix
Copy link
Collaborator Author

echoix commented Oct 23, 2023

/build

Command run output
Build command workflow started.
Installing dependencies
Running script ./build.sh

@echoix echoix changed the title Use App::cpm to install perlcritic faster, and clean cache Use App::cpm to install perlcritic faster, and clean .perl-cpm cache Oct 23, 2023
@echoix echoix temporarily deployed to dev October 23, 2023 04:36 — with GitHub Actions Inactive
@echoix echoix temporarily deployed to dev October 23, 2023 04:36 — with GitHub Actions Inactive
@echoix
Copy link
Collaborator Author

echoix commented Oct 23, 2023

/build

Command run output
Build command workflow started.
Installing dependencies
Running script ./build.sh
Build command workflow completed updating files.

@echoix echoix force-pushed the optimize-install-perlcritic branch from ed1a7b7 to 480342f Compare October 23, 2023 04:50
@echoix echoix temporarily deployed to dev October 23, 2023 04:51 — with GitHub Actions Inactive
@echoix echoix temporarily deployed to dev October 23, 2023 04:51 — with GitHub Actions Inactive
@echoix echoix temporarily deployed to dev October 23, 2023 11:29 — with GitHub Actions Inactive
@echoix echoix temporarily deployed to dev October 23, 2023 11:29 — with GitHub Actions Inactive
Copy link
Member

@nvuillam nvuillam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks :)

@nvuillam nvuillam merged commit ee201af into oxsecurity:main Oct 23, 2023
7 checks passed
@echoix echoix deleted the optimize-install-perlcritic branch October 23, 2023 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file docker Pull requests that update Docker code performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants