Limit test data excluded and support HARFBUZZ_SYS_NO_PKG_CONFIG better #142

spl · 2019-04-02T14:45:22Z

This is an alternative to #138 that builds on the work there.
I added better support for HARFBUZZ_SYS_NO_PKG_CONFIG being 0 or 1.
I expanded the Travis-CI test matrix to test the above as well as add osx.

This change is


        Exclude only test font data


        Support HARFBUZZ_SYS_NO_PKG_CONFIG=0 or 1


        Expand test matrix

* HARFBUZZ_SYS_NO_PKG_CONFIG=0 and 1 * harfbuff-sys package * osx

spl · 2019-04-02T17:56:04Z

I don't understand the error.

On the PR build (HARFBUZZ_SYS_NO_PKG_CONFIG="1"):

$ cargo package --manifest-path=harfbuzz-sys/Cargo.toml
error: 1 files in the working directory contain changes that were not yet committed into git:

harfbuzz/src/hb-ot-shape-complex-use-machine.hh

to proceed despite this, pass the `--allow-dirty` flag

The command "cargo package --manifest-path=harfbuzz-sys/Cargo.toml" exited with 101.

On my build of the same:

$ cargo package --manifest-path=harfbuzz-sys/Cargo.toml
   Packaging harfbuzz-sys v0.3.1 (/home/travis/build/spl/rust-harfbuzz/harfbuzz-sys)
   Verifying harfbuzz-sys v0.3.1 (/home/travis/build/spl/rust-harfbuzz/harfbuzz-sys)
    Updating crates.io index
   Compiling cc v1.0.32
   Compiling pkg-config v0.3.14
   Compiling libc v0.2.51
   Compiling cmake v0.1.37
   Compiling servo-freetype-sys v4.0.5
   Compiling harfbuzz-sys v0.3.1 (/home/travis/build/spl/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1)
   Compiling freetype v0.4.1
    Finished dev [unoptimized + debuginfo] target(s) in 58.07s
The command "cargo package --manifest-path=harfbuzz-sys/Cargo.toml" exited with 0.

Is this a Travis-CI cache issue of some sort?

spl · 2019-04-02T17:59:06Z

The PR is on travis-ci.com, and mine is on travis-ci.org. Does that make a difference?

jdm · 2019-04-02T18:26:45Z

That's the same error that I saw on my PR for one of the CI builds (the other passed). I don't know what to make of it.


        Add ls and git status to debug

spl · 2019-04-03T06:27:16Z

At least now something's happening on both travis-ci.com and travis-ci.org:

https://travis-ci.com/servo/rust-harfbuzz/jobs/189894765 (HARFBUZZ_SYS_NO_PKG_CONFIG="1"):

$ cargo package --manifest-path=harfbuzz-sys/Cargo.toml
   Packaging harfbuzz-sys v0.3.1 (/home/travis/build/servo/rust-harfbuzz/harfbuzz-sys)
   Verifying harfbuzz-sys v0.3.1 (/home/travis/build/servo/rust-harfbuzz/harfbuzz-sys)
    Updating crates.io index
   Compiling cc v1.0.34
   Compiling pkg-config v0.3.14
   Compiling libc v0.2.51
   Compiling cmake v0.1.37
   Compiling servo-freetype-sys v4.0.5
   Compiling harfbuzz-sys v0.3.1 (/home/travis/build/servo/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1)
   Compiling freetype v0.4.1
    Finished dev [unoptimized + debuginfo] target(s) in 57.88s
error: failed to verify package tarball
Caused by:
  Source directory was modified by build.rs during cargo publish. Build scripts should not modify anything outside of OUT_DIR. Modified file:
/home/travis/build/servo/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1/harfbuzz/src/hb-ot-shape-complex-use-machine.hh
To proceed despite this, pass the `--no-verify` flag.
The command "cargo package --manifest-path=harfbuzz-sys/Cargo.toml" exited with 101.

https://travis-ci.org/spl/rust-harfbuzz/jobs/515032933 (HARFBUZZ_SYS_NO_PKG_CONFIG="0"):

$ cargo package --manifest-path=harfbuzz-sys/Cargo.toml
   Packaging harfbuzz-sys v0.3.1 (/home/travis/build/spl/rust-harfbuzz/harfbuzz-sys)
   Verifying harfbuzz-sys v0.3.1 (/home/travis/build/spl/rust-harfbuzz/harfbuzz-sys)
    Updating crates.io index
   Compiling cc v1.0.34
   Compiling pkg-config v0.3.14
   Compiling libc v0.2.51
   Compiling cmake v0.1.37
   Compiling servo-freetype-sys v4.0.5
   Compiling harfbuzz-sys v0.3.1 (/home/travis/build/spl/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1)
   Compiling freetype v0.4.1
    Finished dev [unoptimized + debuginfo] target(s) in 58.55s
error: failed to verify package tarball
Caused by:
  Source directory was modified by build.rs during cargo publish. Build scripts should not modify anything outside of OUT_DIR. Modified file:
/home/travis/build/spl/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1/harfbuzz/util/ansi-print.cc
To proceed despite this, pass the `--no-verify` flag.
The command "cargo package --manifest-path=harfbuzz-sys/Cargo.toml" exited with 101.


        RUST_LOG=trace cargo package


        Reduce stderr output with a grep


        ls after cargo package

spl · 2019-04-03T08:07:24Z

Well, at least I can confirm that the last modified file time does change. Not yet sure if that's useful, but it seems to imply that cargo package is failing for a valid reason.

[2019-04-03T08:00:55Z TRACE cargo::sources::path] last modified file /home/travis/build/servo/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1: 1554278454.221515537s
...
[2019-04-03T08:01:50Z TRACE cargo::sources::path] last modified file /home/travis/build/servo/rust-harfbuzz/target/package/harfbuzz-sys-0.3.1: 1554278467.852523536s

1554278467.852523536 - 1554278454.221515537 = 13.63100791 seconds


        ls -lrt --full-time


        Hack into build.rs: ls out_dir


        cargo package -vv


        Also output for pkg-config


        ls properly in the target dir


        Remove extra ls


        ls harfbuzz/util


        diff -r


        diff || true


        git status


        comment out cargo build to speed up Travis

spl · 2019-04-03T11:45:52Z

Now, when I see:

error: failed to verify package tarball
Caused by:
Source directory was modified by build.rs during cargo publish.
Build scripts should not modify anything outside of OUT_DIR.
Modified file:
.../target/package/harfbuzz-sys-0.3.1/harfbuzz/src/hb-ot-shape-complex-myanmar-machine.hh

I can confirm that its file modification time changed (and no other non-Makefile modification time changed):

-[harfbuzz-sys 0.3.1] -rw-rw-r-- 1 travis travis  12409 2019-04-03 11:36:22.227032921 +0000 hb-ot-shape-complex-myanmar-machine.hh
+[harfbuzz-sys 0.3.1] -rw-rw-r-- 1 travis travis  12409 2019-04-03 11:36:36.647224654 +0000 hb-ot-shape-complex-myanmar-machine.hh

and no content changed:

$ diff -r harfbuzz-sys/harfbuzz target/package/harfbuzz-sys-0.3.1/harfbuzz
Only in harfbuzz-sys/harfbuzz/test/fuzzing: fonts
Only in harfbuzz-sys/harfbuzz/test/shaping/data/aots: fonts
Only in harfbuzz-sys/harfbuzz/test/shaping/data/in-house: fonts
Only in harfbuzz-sys/harfbuzz/test/shaping/data/text-rendering-tests: fonts
Only in harfbuzz-sys/harfbuzz/test/subset/data: fonts

Unfortunately, this particular error and every other one appears nondeterministically.


        rebuild

spl · 2019-04-03T12:02:15Z

Just to prove my point, triggering a rebuild in 4a6d69a caused the error to go away.


        ls > file, diff file


        rebuild

spl · 2019-04-03T12:53:28Z

I'm at a loss. Sometimes, it's util/ansi-print.cc, as in 3b148db. Other times, it's src/hb-ot-shape-complex-myanmar-machine.h or src/hb-ot-shape-complex-use-machine.hh. Sometimes, it's HARFBUZZ_SYS_NO_PKG_CONFIG=0, other times it's HARFBUZZ_SYS_NO_PKG_CONFIG=1. And, finally, sometimes, there's no issue at all. 😱


        sudo pip3

fasihrana · 2019-04-10T10:11:41Z

This just caused an issue for me publishing my own dependent crate to crates.io. I don't see an issue when building binaries, but this shows up in cargo package on fasihrana/skryn

spl · 2019-04-10T11:12:43Z

This just caused an issue for me publishing my own dependent crate to crates.io. I don't see an issue when building binaries, but this shows up in cargo package on fasihrana/skryn

@fasihrana I think you're referring to #137, which was fixed in #144, and that was merged to master 2 days ago. I think @jdm is going to make a release soon.

This particular PR started as a fix for #137 but has since become an attempt to figure out yet another, more elusive issue with cargo package.

fasihrana · 2019-04-10T11:22:49Z

@fasihrana I think you're referring to #137, which was fixed in #144, and that was merged to master 2 days ago. I think @jdm is going to make a release soon.

Thank you. I'll update my crate once that is released.


        Use ffsend instead of send-cli


        ffsend -q


        sed: /bin/true


        Revert sed: /bin/true

Apparently, sed is required by configure.


        reduce touch'd files to configure and Makefile.in


        touch config.h.in


        touch aclocal.m4


        touch -r configure


        fix touch -r


        touch configure.ac


        touch hb-version.h


        touch m4/*.m4


        touch everything


        touch Makefile.* hb-version.*

spl · 2019-04-16T13:26:43Z

While this was a interesting (and challenging) experiment to determine why sporadic and spurious errors were appearing when running cargo package on Travis, I can no longer reproduce these errors in the test matrix of #146. Since #144 already included the key part of this PR and #146 has a better setup, I'm closing this.

spl · 2019-04-16T13:51:09Z

Correction: The cargo package errors seen here are still apparent in #146. I unknowingly hid them behind after_script, a part of the job lifecycle whose errors do not affect the build status. That said, there is still nothing in this PR that warrants keeping it open.


        Avoid file changes during harfbuzz static build

cargo package sporadically complains about file modifications outside the OUT_DIR. See servo#142 for an experiment with chasing down that issue, which can involve different files changed at different times. The build script is written to build out-of-tree in the OUT_DIR, but it seems like 'configure' and/or 'make' still modifies the source tree. This change involves creating a temporary BUILD_DIR in the OUT_DIR, copying the source files there, building, installing into the OUT_DIR, and removing the BUILD_DIR.


        Avoid source changes during static build

cargo package sporadically complains about file modifications outside the OUT_DIR. See servo#142 for an experiment with chasing down that issue, which can involve different files changed at different times. The build script is written to build out-of-tree in the OUT_DIR, but `configure` and/or `make` still modifies the source tree. This change involves creating a temporary BUILD_DIR in the OUT_DIR, copying the source files there, building, installing into the OUT_DIR, and, finally, removing the BUILD_DIR.


        Avoid source changes during static build

cargo package sporadically complains about file modifications outside the OUT_DIR. See servo#142 for an experiment with chasing down that issue, which can involve different files changed at different times. The build script is written to build out-of-tree in the OUT_DIR, but `configure` and/or `make` still modifies the source tree. This change involves creating a temporary BUILD_DIR in the OUT_DIR, copying the source files there, building, installing into the OUT_DIR, and, finally, removing the BUILD_DIR.


        Avoid source changes during static build

`cargo package` sporadically complains about file modifications outside `OUT_DIR`. See servo#142 for an experiment with chasing down that issue, which can involve different files changed at different times. The build script is written to build out-of-tree in `OUT_DIR`, but `configure` and/or `make` still modify the source tree. This change involves creating a temporary `BUILD_DIR` in `OUT_DIR`, copying the source files there, building, installing into `OUT_DIR`, and, finally, removing `BUILD_DIR`.


        Auto merge of #147 - spl:fix-build, r=jdm

Avoid source changes during static build `cargo package` sporadically complains about file modifications outside `OUT_DIR`. See #142 for an experiment with chasing down that issue, which can involve different files changed at different times. The build script is written to build out-of-tree in `OUT_DIR`, but `configure` and/or `make` still modify the source tree. This change involves creating a temporary `BUILD_DIR` in `OUT_DIR`, copying the source files there, building, installing into `OUT_DIR`, and, finally, removing `BUILD_DIR`.  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/rust-harfbuzz/147)

jdm and others added 3 commits Mar 27, 2019

Exclude only test font data

Loading status checks…

7234ca7

Support HARFBUZZ_SYS_NO_PKG_CONFIG=0 or 1

573ad88

Expand test matrix

Loading status checks…

393de23

* HARFBUZZ_SYS_NO_PKG_CONFIG=0 and 1 * harfbuff-sys package * osx

jdm approved these changes Apr 2, 2019

View changes

Add ls and git status to debug

Loading status checks…

da91699

spl added 3 commits Apr 3, 2019

RUST_LOG=trace cargo package

Loading status checks…

e6e8d83

Reduce stderr output with a grep

Loading status checks…

30153ec

ls after cargo package

Loading status checks…

2cdcd4d

spl added 11 commits Apr 3, 2019

ls -lrt --full-time

Loading status checks…

3546ee2

Hack into build.rs: ls out_dir

Loading status checks…

479b565

cargo package -vv

5e7d016

Also output for pkg-config

87e4dff

ls properly in the target dir

Loading status checks…

fce9927

Remove extra ls

d260388

ls harfbuzz/util

Loading status checks…

74818f7

diff -r

Loading status checks…

2b41954

diff || true

Loading status checks…

f25cc2e

git status

Loading status checks…

e89d9fd

comment out cargo build to speed up Travis

Loading status checks…

6a5dd01

rebuild

Loading status checks…

4a6d69a

spl added 2 commits Apr 3, 2019

ls > file, diff file

Loading status checks…

e6f281e

rebuild

Loading status checks…

3b148db

sudo pip3

Loading status checks…

e6b71b2

spl added 13 commits Apr 10, 2019

Use ffsend instead of send-cli

Loading status checks…

286eed0

ffsend -q

3456e8b

sed: /bin/true

Loading status checks…

a2f62bf

Revert sed: /bin/true

Loading status checks…

0c80944

Apparently, sed is required by configure.

reduce touch'd files to configure and Makefile.in

Loading status checks…

552b0a4

touch config.h.in

Loading status checks…

a678eac

touch aclocal.m4

Loading status checks…

6a30574

touch -r configure

Loading status checks…

794477f

fix touch -r

Loading status checks…

188d6ff

touch configure.ac

Loading status checks…

59c7345

touch hb-version.h

Loading status checks…

502f6c3

touch m4/*.m4

Loading status checks…

0934536

touch everything

Loading status checks…

e796611

jdm mentioned this pull request Apr 16, 2019

Build harfbuzz crate in CI. #138

Closed

touch Makefile.* hb-version.*

Loading status checks…

58c0aee

spl closed this Apr 16, 2019

spl deleted the spl:cargo-package branch Apr 16, 2019

spl mentioned this pull request Apr 16, 2019

Create test matrix for Travis-CI #146

Merged

spl mentioned this pull request Apr 17, 2019

Avoid source changes during static build #147

Merged

servo / rust-harfbuzz

Limit test data excluded and support HARFBUZZ_SYS_NO_PKG_CONFIG better #142

Limit test data excluded and support HARFBUZZ_SYS_NO_PKG_CONFIG better #142

spl commented Apr 2, 2019 •

edited by larsbergstrom

spl commented Apr 2, 2019

spl commented Apr 2, 2019

jdm commented Apr 2, 2019

spl commented Apr 3, 2019 •

edited

spl commented Apr 3, 2019

spl commented Apr 3, 2019 •

edited

spl commented Apr 3, 2019

spl commented Apr 3, 2019

fasihrana commented Apr 10, 2019

spl commented Apr 10, 2019

fasihrana commented Apr 10, 2019

spl commented Apr 16, 2019

spl commented Apr 16, 2019

servo / rust-harfbuzz

Join GitHub today

Limit test data excluded and support HARFBUZZ_SYS_NO_PKG_CONFIG better #142

Limit test data excluded and support HARFBUZZ_SYS_NO_PKG_CONFIG better #142

Conversation

spl commented Apr 2, 2019 • edited by larsbergstrom

spl commented Apr 2, 2019

spl commented Apr 2, 2019

jdm commented Apr 2, 2019

spl commented Apr 3, 2019 • edited

spl commented Apr 3, 2019

spl commented Apr 3, 2019 • edited

spl commented Apr 3, 2019

spl commented Apr 3, 2019

fasihrana commented Apr 10, 2019

spl commented Apr 10, 2019

fasihrana commented Apr 10, 2019

spl commented Apr 16, 2019

spl commented Apr 16, 2019

spl commented Apr 2, 2019 •

edited by larsbergstrom

spl commented Apr 3, 2019 •

edited

spl commented Apr 3, 2019 •

edited