vector: 0.10.0 -> 0.12.1

[thoughtpolice]

As a minor regression, LevelDB support is currently compiled out. This is due to a few changes in the build infrastructure that now causes leveldb to be vendored when it shouldn't be, but should be fixable, I think.

This has to hit staging because Vector now requires Rust 1.50, due to it stabilizing some library APIs it uses.

Darwin is currently untested.

This supersedes both #113849 and #107557.

/cc @happysalada @rmcgibbo

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[rmcgibbo]

(I hid the bot comment -- looks like an OOM or something)

[happysalada]

Result of nixpkgs-review pr 116017 run on x86_64-darwin 1

1 package failed to build:

• vector

[happysalada]

I'm not really sure what it's due to. Here is the error i get

error: --- Error --- nix-daemon
output '/nix/store/3j0j0i9n8dqi8pj8zn73mc62708rakj5-bootstrap-stage2-stdenv-darwin' is not allowed to refer to the following paths:
  /nix/store/qvv2xw5da22lfgbj1zq4hhvh286i74f7-brotli-1.0.9-lib

It looks like a darwin specific thing.

[SuperSandro2000]

I'm not really sure what it's due to. Here is the error i get

error: --- Error --- nix-daemon
output '/nix/store/3j0j0i9n8dqi8pj8zn73mc62708rakj5-bootstrap-stage2-stdenv-darwin' is not allowed to refer to the following paths:
  /nix/store/qvv2xw5da22lfgbj1zq4hhvh286i74f7-brotli-1.0.9-lib

It looks like a darwin specific thing.

Fixed by #115498

[happysalada]

Just as a note, it looks like 12.1 has been released.
https://github.com/timberio/vector/releases/tag/v0.12.1

[happysalada]

Result of nixpkgs-review pr 116017 run on x86_64-darwin 1

1 package failed to build:

• vector

[happysalada]

I've got totally unrelated failures again

builder for '/nix/store/02r3nw3cmvr70r7lp3p4xjjl5jbvncbm-libtiff-4.2.0.drv' failed with exit code 2; last 10 log lines:

  The following tests FAILED:
        18 - tiffdump-miniswhite-1c-1b (Failed)
        19 - tiffinfo-minisblack-1c-16b (Failed)
       20 - tiffcp-split (Failed)
     21 - tiffcp-split-join (Failed)
        22 - tiff2pdf-miniswhite-1c-1b (Failed)
       108 - testtypes (Subprocess aborted)
  Errors while running CTest
  make: *** [Makefile:149: test] Error 8
error: --- Error --- nix-daemon
builder for '/nix/store/3gnlw2rhp5211sysi4x8ildpslblyk3r-python3.8-lxml-4.6.2.drv' failed with exit code 1; last 10 log lines:
  running build_ext
  building 'lxml.etree' extension
  creating build/temp.macosx-10.6-x86_64-3.8
  creating build/temp.macosx-10.6-x86_64-3.8/src
  creating build/temp.macosx-10.6-x86_64-3.8/src/lxml
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc/lxml -Isrc/lxml/includes -I/nix/store/180s86m4qgd4mhybxx1y2fh3vzpbcrgs-libxml2-2.9.10-dev/include/libxml2 -I/nix/store/fwmrca0zvgijjy5hygv07pr21gfpv2zp-libxslt-1.1.34-dev/include -Isrc -I/nix/store/xrrf9gx1lz45cj6ca2xb7bak6p8b8rkj-python3-3.8.8/include/python3.8 -c src/lxml/etree.c -o build/temp.macosx-10.6-x86_64-3.8/src/lxml/etree.o -w -flat_namespace
  clang -bundle -undefined dynamic_lookup -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.2.10/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.3/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.19/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.34.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1j/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.2.10/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.3/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.19/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.34.1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1j/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib build/temp.macosx-10.6-x86_64-3.8/src/lxml/etree.o -L/nix/store/h3g32s4q348irqb59vvklkbs15dmc9vg-libxml2-2.9.10/lib -L/nix/store/yppiyxw1vd553ihxxbnwlzi6033p2kld-libxslt-1.1.34/lib -L/nix/store/xrrf9gx1lz45cj6ca2xb7bak6p8b8rkj-python3-3.8.8/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.macosx-10.6-x86_64-3.8/lxml/etree.cpython-38-darwin.so -isysroot
  clang-7: error: no such file or directory: 'x86_64-apple-darwin'
  clang-7: warning: no such sysroot directory: '-target' [-Wmissing-sysroot]
  error: command 'clang' failed with exit status 1

[thoughtpolice]

@ofborg build vector

[happysalada]

The build takes 7 hours on my machine, I'll try to remember to trigger it before i go to sleep tonight.

[thoughtpolice]

Part of that is definitely just the use of staging, though... there's no available mac that can really make those rebuilds tolerable outside the Pro range.

I think we should probably just go ahead and let this percolate to master naturally, and issue darwin fixes as needed.

I still haven't figured out the LevelDB thing.

[thoughtpolice]

Merging this to staging. Filing a bug about the regressions.

[thoughtpolice]

We can follow up in #116574

[happysalada]

I completely agree, I was about to say that most probably very few people would use vector on a mac.
I personally only use it on production. I would be surprised to see much need for this for darwin.
Thanks for merging!

[rmcgibbo]

Great! I will use this in the next iteration of r-rmcgibbo (I had previously switched to using the upstream-released binaries). Awesome.

[happysalada]

@thoughtpolice is there a way we can merge this into master?
At the moment vector 0.10 fails to build, so there is no way to use vector on nixos.
Since it's (in my opinion) the primary platform where vector would be used, it would make it available again to merge it.

[thoughtpolice]

If there's a way to enable unstable Rust features in master, then it might be possible (again, this is due to the Rust 1.50 requirement, which I suspect is part of the reason 0.12 took so long to release in the first place.) But I couldn't find any cases of anyone doing this, so I came to the conclusion it might even involve a global rebuild of all Rust packages to do so. But at that point just putting it in staging is probably just as good.

I can't give a good prediction on when staging will be merged into master. It seems like staging-next was merged last week, so it might be possible to do an incremental merge with just a few updates including vector, but I'd have to look over the exact diff to be sure.

[happysalada]

Thanks for the explanation!

[happysalada]

It seems there is a new release of vector, would you also want to merge this on staging? Or wait until staging is merged on master before doing a PR?

[SuperSandro2000]

It seems there is a new release of vector, would you also want to merge this on staging? Or wait until staging is merged on master before doing a PR?

It is no problem to update staging again.