Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce disk usage of thirdparty build directories #1578

Merged
merged 3 commits into from
Feb 19, 2023

Conversation

benoit-pierre
Copy link
Contributor

@benoit-pierre benoit-pierre commented Feb 5, 2023

  • shallower clones
  • sparse checkouts for the initial repos
  • support not fetching/setting up any submodules
  • disable all OpenSSL submodules (not used)
  • only fetch/checkout the MuPDF submodules that are needed

Stats after building 3 versions (emulator debug & release versions, kindlepw2 release version):

master PR
thirdparty/*/build/git_checkout 2.6G 296M
thirdparty/*/build/x86_64-pc-linux-gnu 3.2G 1.5G
thirdparty/*/build/x86_64-pc-linux-gnu-debug 3.6G 2.0G
thirdparty/*/build/arm-kindlepw2-linux-gnueabi 3.5G 1.8G
thirdparty/*/build 13.0G 5.5G

Still high, but much better.


This change is Reviewable

@@ -69,79 +60,76 @@ if(should_clone)
if(error_code)
message(FATAL_ERROR \"Failed to clone repository: '${git_repository}'\")
endif()
execute_process(
COMMAND \"${git_EXECUTABLE}\" -C \"${clone_checkout}\" sparse-checkout init
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't supported yet. But thanks for providing the update.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: the code should be robust to sparse-checkout not being supported (exit codes are ignored for that reason).

Copy link
Member

@Frenzie Frenzie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not about this PR which is exactly according to plan but blocked until the Android image is updated.

@Frenzie
Copy link
Member

Frenzie commented Feb 19, 2023

It's a bit unclear to me how you managed to make CircleCI not run. Do you have any idea?

@benoit-pierre benoit-pierre reopened this Feb 19, 2023
@Frenzie Frenzie merged commit bd895c8 into koreader:master Feb 19, 2023
@Frenzie
Copy link
Member

Frenzie commented Feb 19, 2023

Eh well, I think it's fine with a quick try, though it's lacking the peace of mind of having a multitude of configurations checked.

Now running on master: https://github.com/koreader/koreader-base/runs/11450672853

@benoit-pierre
Copy link
Contributor Author

It's a bit unclear to me how you managed to make CircleCI not run. Do you have any idea?

No idea, but from past experience Circle CI can be a bit funky.

@benoit-pierre benoit-pierre deleted the trimming_the_fat branch February 19, 2023 17:50
@benoit-pierre
Copy link
Contributor Author

Have you considered converting those jobs to use Github Actions?

@benoit-pierre
Copy link
Contributor Author

No looking good on master...

@benoit-pierre
Copy link
Contributor Author

I'm looking into it, seems like mupdf is missing some variables (e.g. SYS_LIBJPEG_LIBS).

@Frenzie
Copy link
Member

Frenzie commented Feb 19, 2023

Have you considered converting those jobs to use Github Actions?

GitHub Actions is unfortunately not fit for purpose. See #1442 (comment) and #1464.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants