Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider minifying json shipped in package #1967

Closed
SimonSchick opened this issue Nov 1, 2022 · 2 comments · Fixed by #2315
Closed

Consider minifying json shipped in package #1967

SimonSchick opened this issue Nov 1, 2022 · 2 comments · Fixed by #2315

Comments

@SimonSchick
Copy link

tl;dr we are deploying the SDK to aws lambdas and ran into a scenario where our primary layer became too large, initially we just stripped individual json files from discovery_cache/documents since this is by far the largest culprit (according to ncdu).

I also noticed that all of these definitions are actually pretty printed json.

I'm suggesting 2 things:

  1. Ship a utility to select (and delete other) json definitions that the user wants to use in a deployment.
  2. Minify all json contents (see trivial script below), this reduced the package size (on disk) by approximately 20mb (and probably also improves load times although I did not test for that).
fs.readdirSync('.').forEach(f => {
  const data = JSON.parse(fs.readFileSync(f, 'utf8'));
  fs.writeFileSync(f, JSON.stringify(data));
  console.log('processed', f);
});

Environment details

  • OS type and version: irrelevant
  • Python version: 3.9
  • pip version: 22.2.2
  • google-api-python-client version: 2.64.0

Steps to reproduce

  1. Install package
  2. 72mb via ncdu

Code example

n/a

Stack trace

n/a

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

@tgbugs
Copy link

tgbugs commented Mar 20, 2023

A related issue on the aws side: boto/botocore#2365. Their solution was to gzip the json files.

@jorenham
Copy link
Contributor

At the moment the json files take 81.3MB on disk, which is a significant portion of my docker images.

jorenham added a commit to jorenham/google-api-python-client that referenced this issue Apr 1, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
parthea pushed a commit that referenced this issue May 24, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: #1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue May 24, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue May 26, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue May 30, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue Jun 13, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue Jul 5, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue Jul 18, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
jorenham added a commit to jorenham/google-api-python-client that referenced this issue Sep 20, 2023
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: googleapis#1967
parthea added a commit that referenced this issue Jan 8, 2024
Changed the discovery artifact JSON indentation from 2 to 0, in
order to reduce the size by several megabytes.

Refs: #1967

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants