Broken distribution files for 2.4.0

[atsampson]

The distribution files for 2.4.0 on freedesktop.org that have appeared today seem to be broken - the .sha256 file is empty, and the .tar.bz2 is only 14 bytes.

[ebraminio]

No GitHub upload also.

And what has happened between 2.2.0 and 2.3.0, that looks weird.

[dscorbett]

Also, some “REPLACEME”s were missed.

[behdad]

Umm.. I've been without a Linux laptop for two months now. Tried making a release on a Mac and here we go...

I'll fix up today.

[behdad]

(Also why I have not been making win32 bundles. If anyone wants to add those, I can upload them. If anyone wants to take over release engineering, I'd appreciate that!)

[behdad]

I don't know what's wrong with my Mac setup, but I tried "make dist" again, and still get the 14-byte tarball. Can someone build the tarball for 2.4.0 commit and send it to me to sign and upload?

[behdad]

@ebraminio @dscorbett I'll wait for help from either of you. Thanks.

[behdad]

Or feel free to make a 2.4.1 release.

[khaledhosny]

I can make the tarball, but you didn’t push a tag for 2.4.0

[behdad]

I can make the tarball, but you didn’t push a tag for 2.4.0

Gah. Somehow git doesn't like to sign, even though I've set up my gpg:

error: gpg failed to sign the data
error: unable to sign the tag
The tag message has been left in .git/TAG_EDITMSG

Do the tag -s and push as well? Thanks Khaled.

[behdad]

Okay, figured out. Pushed tag.

[khaledhosny]

Tarball sent.

[ryandesign]

I don't know what's wrong with my Mac setup, but I tried "make dist" again, and still get the 14-byte tarball.

@behdad The .tar.bz2 files was 14 bytes; the .tar file inside was 0 bytes.

I tried make dist on my Mac (macOS 10.13.6, Xcode 9.4.1, dependencies installed with MacPorts) and it produced a reasonable .tar.bz2 file.

Maybe I can help you sort out what went wrong on your machine. Did you get any errors about the harfbuzz-2.4.0 directory not getting created, or any errors from tar while creating the tarball?

[behdad]

And what has happened between 2.2.0 and 2.3.0, that looks weird.

Oh. Umm yeah. I think @blueshade7 added the entirety of SourceHanSans to the repo! We should remove it. Tarball size grew from 4MB to 17MB!

@blueshade7 Please replace that with a subset. Thanks.

[behdad]

Should be fixed now. But yeah, we should remove that huge font. I wish there was an easy way to clean it up from the repo as well, but there isn't without rewriting history....

[behdad]

Maybe I can help you sort out what went wrong on your machine. Did you get any errors about the harfbuzz-2.4.0 directory not getting created, or any errors from tar while creating the tarball?

Nope. It "succeeds". But this does indeed look suspicous:

	  dist-hook
cp: ./harfbuzz.pdf: No such file or directory
make[5]: [dist-hook] Error 1 (ignored)
cp: ./harfbuzz.types: No such file or directory
make[5]: [dist-hook] Error 1 (ignored)
/Applications/Xcode_10.1_fb.app/Contents/Developer/usr/bin/make  \
	  top_distdir="harfbuzz-2.4.0" distdir="harfbuzz-2.4.0" \
	  dist-hook
chmod -R a-s harfbuzz-2.4.0
test -n "" \
	|| find "harfbuzz-2.4.0" -type d ! -perm -755 \
		-exec chmod u+rwx,go+rx {} \; -o \
	  ! -type d ! -perm -444 -links 1 -exec chmod a+r {} \; -o \
	  ! -type d ! -perm -400 -exec chmod a+r {} \; -o \
	  ! -type d ! -perm -444 -exec /bin/sh /Users/behdad/hb/install-sh -c -m a+r {} {} \; \
	|| chmod -R a+r "harfbuzz-2.4.0"
tardir=harfbuzz-2.4.0 && false | BZIP2=${BZIP2--9} bzip2 -c >harfbuzz-2.4.0.tar.bz2
if test -d "harfbuzz-2.4.0"; then find "harfbuzz-2.4.0" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "harfbuzz-2.4.0" || { sleep 5 && rm -rf "harfbuzz-2.4.0"; }; else :; fi

Running "false" instead of tar?! No idea why. I definitely have some tar:

$ tar
tar: Must specify one of -c, -r, -t, -u, -x

[behdad]

brew install gnu-tar and rerunning configure didn't help either.

[behdad]

checking how to create a ustar tar archive... none
So, automake bug to NOT fail...

[khaledhosny]

Sending new tarball… (not the fastest internet connection 😄 to upload a 23MB attachment).

[behdad]

checking how to create a ustar tar archive... none
So, automake bug to NOT fail...

Apparently because:
configure:3521: checking whether UID '407006440' is supported by ustar format configure:3527: result: no configure:3531: checking whether GID '1876110778' is supported by ustar format configure:3537: result: no

[atsampson]

Can I recommend doing make distcheck instead of make dist as part of your release process in the future? distcheck confirms that the tarball can be used to compile the project and run the test suite, which is a useful sanity check for various kinds of problems.

[behdad]

Can I recommend doing make distcheck instead of make dist as part of your release process in the future? distcheck confirms that the tarball can be used to compile the project and run the test suite, which is a useful sanity check for various kinds of problems.

Right. We normally do that. We have a full document on this: https://github.com/harfbuzz/harfbuzz/blob/master/RELEASING.md

It was my fault. distcheck was mysteriously failing, and since I have had to debug and install multiple things for just dist to work, since dist worked without error I accepted it, knowing that in general we are distcheck-clean. Sigh. Shouldn't happen again. I'll switch back to Linux soonl

[ryandesign]

checking how to create a ustar tar archive... none
So, automake bug to NOT fail...

Well, it's documented behavior, not a bug: "configure knows several ways to construct these formats. It will not abort if it cannot find a tool up to the task (so that the package can still be built), but make dist will fail."

Apparently because:
configure:3521: checking whether UID '407006440' is supported by ustar format configure:3527: result: no configure:3531: checking whether GID '1876110778' is supported by ustar format configure:3537: result: no

Looks like IBM says IDs above 2,097,151 should not be used with ustar format, and Oracle says IDs above 60,000 should not be used on Solaris. Perhaps the reason why these IDs were used on your system is that your Mac is connected to a directory server which gives out such large IDs? If you could create a local Mac user account, not connected to your directory server, that should have a normal small ID which would work.

I'd say you should contact your network administrator and ask them to fix their server to provide smaller IDs, but it seems like network administrators aren't very interested in proving working services. We recently had a report at MacPorts from a user whose company's directory server gave him a username containing a space; hilarity ensued.

[behdad]

checking how to create a ustar tar archive... none
So, automake bug to NOT fail...

Well, it's documented behavior, not a bug: "configure knows several ways to construct these formats. It will not abort if it cannot find a tool up to the task (so that the package can still be built), but make dist will fail."

But "make dist" does NOT fail for me.

[behdad]

And yes, this is a company laptop. I don't have control over the ids.

[spl]

The current 2.4.0 tarball is very different from the repository at the 2.4.0 tag. When I do a diff (diff -uar) of the two src directories, I get a 2937-line file that is 98KB (which includes things like Only in harfbuzz-repo/src: hb-dsalgs.hh). Also, the tarball didn't build on AppVeyor with one of the same configurations in your appveyor.yml (servo/rust-harfbuzz#141).

[behdad]

Uploaded new one... This release is cursed :).