Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop packaging node_modules in release tarballs #15273

Merged
merged 3 commits into from
Apr 9, 2021

Conversation

silverwind
Copy link
Member

@silverwind silverwind commented Apr 4, 2021

  • Don't package node_modules in tarballs, they are not cross-platform anymore and npm cache should not be messed with directly. Instead, require an internet connection to rebuild the UI, which is not necessary in the general use case because prebuilt UI files are shipped in the public directory.
  • Simplify the fomantic build and make the target phony. We don't need anything more for something that is rarely ran.
  • Use regular tar again to build tarballs and add variable for excludes
  • Disable annoying npm update notifications

Fixes: #14578
Fixes: #15256
Fixes: #15262

@silverwind silverwind added the topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile label Apr 4, 2021
@silverwind silverwind added this to the 1.15.0 milestone Apr 4, 2021
@silverwind silverwind force-pushed the no-tar-rebuilds branch 2 times, most recently from c6ee793 to 3803d24 Compare April 4, 2021 10:05
@silverwind silverwind mentioned this pull request Apr 4, 2021
Makefile Outdated Show resolved Hide resolved
@GiteaBot GiteaBot added lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Apr 4, 2021
Makefile Outdated Show resolved Hide resolved
@6543
Copy link
Member

6543 commented Apr 6, 2021

@silverwind is there a reason to dorp npm-cache?

I thought this was to be able building frontend without internet conection

@silverwind
Copy link
Member Author

Yes, I think npm cache is not something that is meant to be interfaced with directly and doing so introduces a lot of unnecessary complexity that is also prone to break on future npm versions. I'm not sure how others feel about this but for me it has crossed the threshold of being maintainable in the long run.

@6543
Copy link
Member

6543 commented Apr 6, 2021

I'm ok with it, as long as make release-sources create a tarball able to build gitea without internet

@CL-Jeremy
Copy link
Contributor

The npm cache is strictly a cache: it should not be relied upon as a persistent and reliable data store for package data. npm makes no guarantee that a previously-cached piece of data will be available later, and will automatically delete corrupted contents.

Yeah, I actually know this. The fact npm 7 changes so much as to convert relative path to absolute path (!) has really made me rethink about this. Also, I revisited Gogs and saw them still relying on lessc in $PATH, which is even less friendly for new contributors.

@CL-Jeremy
Copy link
Contributor

@6543 If you are excluding ./data, you might as well exclude ./gitea and ./log. I did not bother to include these since I thought this target is "hidden" from most users (not listed in make help).

Also the second apt -y install could be changed to apt-get install -y for consistency.

@silverwind
Copy link
Member Author

silverwind commented Apr 6, 2021

build gitea without internet

TAGS="bindata" make build must of course still work from a source tarball while offline, it should just skip over the frontend target because WEBPACK_DEST will have a newer timestamp than its dependencies. I will verify this later.

The frontend target actually checks for node.js to be present which in this case would not be necessary, so I think that could be fixed as well by moving the node check.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Apr 6, 2021
@silverwind
Copy link
Member Author

silverwind commented Apr 6, 2021

Hold on a bit with merging here, I will do a few more corrections as discussed.

@6543
Copy link
Member

6543 commented Apr 6, 2021

well for me it is ok ... still need one lgtm - will add wip lable you can remove if u are done

@6543 6543 added the pr/wip This PR is not ready for review label Apr 6, 2021
@silverwind
Copy link
Member Author

silverwind commented Apr 6, 2021

Ready again for review.

I used a sub-make call to enable make build to not trigger node-check and node_modules targets via dependencies of WEBPACK_DEST so it is now possible again to build from a tarball without node or internet. I verified like this:

rm -rf /tmp/buildtest && mkdir /tmp/buildtest
make frontend generate release-sources && cp dist/release/gitea-src-master.tar.gz /tmp/buildtest
# disable wifi
cd /tmp/buildtest && tar xf gitea-src-master.tar.gz && TAGS=bindata make build

Also, I added a bunch more tarball excludes for directories that can exist in a development setup, so the locally built tarballs should be close to what the CI builds.

@silverwind silverwind removed the pr/wip This PR is not ready for review label Apr 6, 2021
- Don't package node_modules in tarballs, they are not cross-platform
  anymore and npm cache should not be messed with directly. Instead,
  require an internet connection to rebuild the UI, which is not necessary
  in the general use case because prebuilt UI files are shipped in the
  public directory.
- Simplify the fomantic build and make the target phony. We don't need
  anything more for something that is rarely ran.
- Use regular tar again to build tarballs and add variable for excludes
- Disable annoying npm update notifications

Fixes: go-gitea#14578
Fixes: go-gitea#15256
Fixes: go-gitea#15262
@silverwind
Copy link
Member Author

Rebased and squashed.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Apr 9, 2021
@techknowlogick techknowlogick merged commit 0991f9a into go-gitea:master Apr 9, 2021
@silverwind silverwind deleted the no-tar-rebuilds branch April 9, 2021 05:57
@go-gitea go-gitea locked and limited conversation to collaborators May 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants