Build setup of 1.11.0, use of npm

[stblassitude]

• Gitea version (or commit ref):1.11.0
• Git version:
• Operating system: FreeBSD
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

I'm the maintainer for the FreeBSD port of Gitea, allowing FreeBSD users to simply pkg install gitea.

It seems that with 1.11.0, the CSS and JS files are not included pre-built in the repo anymore, but are built during the Gitea build process, requiring the use of npm install, including network access to download almost a thousand node modules.

The FreeBSD package build system does not allow network access during the build phase; all required files need to be downloaded first using a build-system specific download mechanism from pre-recorded URLs. Among other aspects, this allows the build system to verify cryptographic checksums on files downloaded from potentially unreliable mirrors for source packages.

The FreeBSD ports/package system currently contains no provisions to handle npm install as part of a package build.

This means that with the current state, no FreeBSD package of Gitea 1.11.0 can be produced.

Would you consider adding ./node_modules to the source archive for releases, or provide a single file archive for download of it (matching the release), so that the necessary node modules can be made available to package builds without the necessity to run npm install? I don't think it's necessary to add the directory to the Git repo, only make it part of the release tar.gz. I see this as similar to the vendoring of Go modules.

I have a suspicion that other OS/distributions might have similar restrictions on files that are used during package builds.

[silverwind]

I could see the option of having a separate download for a gitea-$release-node_modules.gzip for this use case. It's important that the extracted node_modules folder's timestamp is newer than package-lock.json, otherwise the build process will try to contact the npm registry, so that file needs to be generated after the main build process.

I do have to wonder if there are more general solutions to this problem. We do use package-lock.json and the dependency packages on npm can be considered immutable, so build should always be exactly reproducible. I know some distributions do package individual npm modules, but I've never been a fan of that.

[guillep2k]

However, not all go modules are vendored. As much as we can include the generated js and css files, I think network access is still required to resolve go dependencies. How was that resolved in previous versions?

[stblassitude]

However, not all go modules are vendored. As much as we can include the generated js and css files, I think network access is still required to resolve go dependencies. How was that resolved in previous versions?

As far as I'm aware, all Go module dependencies are vendored, at least up to 1.10.3.

[lafriks]

True, go modules are included. We could indeed publish node_modules to download folder beside gitea binaries. That could also speedup our own builds with this plugin and s3 configuration: http://plugins.drone.io/meltwater/drone-cache/

[silverwind]

Go modules are vendored currently, but I think the plan is to eventually un-vendor them (Go 1.14 or later), so they too would then need a solution for a "offline" build.

[lafriks]

We can cache them to Gitea dl s3 also then

[lunny]

We can add a sub directory /cache which includes /cache/v1.11.0/go_modules/, /cache/v1.11.0/node_modules/ on dl.gitea.io

[sapk]

I started earlier #9733 it need to be changed to only update cache on master or version branch to limit cache poisonning.

[ecsgh]

Same problem in openSUSE.
I think all build systems does not allow download external files.
All go modules are vendored and includes in source.
But all nodejs packages are not included.
I think nodejs modules must also vendored like go modules.

And please, don't stop to include this modules.
There are needed for building in build systems.
Without that, gitea could no more include in repositories and distributions without extremly more work and "tricks".

[silverwind]

Vendoring node_modules would currently add 36231 files at 276 MB (42M gzipped) to the repo.

I still wonder why distributions need their build systems to be offline.

[sapk]

Maybe a .tar.gz (for each tag) of source code with vendor inside (go and node) is good for you ? This way the build is reproducible from those archive.

[silverwind]

I just checked what Kibana (who is also a heavy node modules user): They don't have node_modules in source control but their release source tarballs do include node_modules.

So I think we just need to update the release process to create a source tarball with modules included. One has to keep in mind that they can be platform-specific when native node.js modules are present, but luckily we don't really have any of those (besides the optional fsevents on macOS), so our node_modules should be generally portable.

[stblassitude]

Including the modules in the release source tarballs would be perfect!

[rnhmjoj]

I still wonder why distributions need their build systems to be offline.

Reproducibility.

[zeripath]

@silverwind - https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/

[sapk]

You should be able to download the archive for the last commit on 1.11 branch on https://dl.gitea.io/gitea/1.11/gitea-src-1.11.tar.gz

The next release 1.11.1 should be released when tagged at the url https://dl.gitea.io/gitea/1.11.1/gitea-src-1.11.1.tar.gz

You can use the signature and the hash that is distributed along the archive.

[sapk]

The CSS and JS builds are also kept inside the public folder so you can skip the nodejs part if you want.

[silverwind]

Gave it a try, TAGS=bindata make build did produce a proper binary and JS/CSS steps were skipped as expected.

There is one minor issues with a error "fatal: not a git repository (or any of the parent directories): .git" seen because the Makefile calls GITEA_VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed 's/^v//') which will of course fail when not in a git checkout.

I'm also thinking we should probably add a explicit make build-offline because the "online" steps are only skipped based on timestamps which seem too brittle to me.

[lunny]

Or we could create a version file before create source tar file. And makefile could read version from the file before read from git. The version file could also tell user which version is the gitea source.

[sapk]

I removed .git from the archive for space but we could simply keep it.
But also the idea of @lunny seems better.

[silverwind]

I'd just parse the version from git, write it to VERSION, gzip, delete VERSION again. Also .gitignore it just in case.

[guillep2k]

I think we should write the version number to a temporary file and ship the sources with that. Then use a make build-offline step as suggested by @silverwind to avoid a dependency of git and reduce the release size.

[silverwind]

The primary reason for a make make build-offline is so it never attempts to go on the network. So if node_modules (or future go modules) are absent, build will fail instead of falling back to network like make build does.

[silverwind]

#10325

[silverwind]

Please reopen this issue. It will be properly resolved once #10325 is landed and backported to 1.11.

[strk]

I'm also hating nodejs dependency, it really sucks, when main reason to follow Gitea (Gogs, actually, when I started) was the niceties of only needing "go" compiler. Please save us from nodejs, what is it needed for, again ?

[silverwind]

The released source tarballs will be able to build without Node.js using a special make target.

what is it needed for, again ?

For webpack, which currently handles JS transpilation and lazy-loading, CSS preprocessing and SVG bundling. Modern frontend development would simply not be possible without it.

[silverwind]

In the next minor 1.11 release, you will be able to use the src tarballs which include prebuilt frontend files so you can run make backend to only built the go code without needing Node.js, which is only required for the make frontend target.