envoy has 33 failing tests when run in an STL debug mode

[jmarantz]

Description:
STL specifies multiple rules which appear to not be enforced in practice but, depending on the plaform, can cause undefined behavior. There's a way to compile so that STL spends extra CPU time to enforce these rules. Obviously you don't go to production like that but it's ultimately a good idea to try to add that to CI. But first the tests need to work, and many of them don't.

[optional Relevant Links:]
http://www.cplusplus.com/reference/stl/

examples

• vector iterators invalidated after a potential resize
• dereferencing end
• incrementing end
• decrementing begin

Repro steps:

blaze test -c dbg //test/... --cxxopt -D_GLIBCXX_DEBUG

Logs:

//test/common/access_log:grpc_access_log_impl_test                       FAILED in 5.2s
//test/common/json:json_loader_test                                      FAILED in 0.8s
//test/common/router:config_impl_test                                    FAILED in 1.5s
//test/common/router:header_formatter_test                               FAILED in 1.2s
//test/common/upstream:cluster_manager_impl_test                         FAILED in 7.2s
//test/common/upstream:upstream_impl_test                                FAILED in 8.6s
//test/config_test:example_configs_test                                  FAILED in 10.0s
//test/integration:access_log_integration_test                           FAILED in 4.4s
//test/integration:ads_integration_test                                  FAILED in 5.5s
//test/integration:cors_filter_integration_test                          FAILED in 1.6s
//test/integration:echo_integration_test                                 FAILED in 10.3s
//test/integration:grpc_json_transcoder_integration_test                 FAILED in 1.6s
//test/integration:header_integration_test                               FAILED in 7.2s
//test/integration:http2_integration_test                                FAILED in 2.5s
//test/integration:http2_upstream_integration_test                       FAILED in 1.8s
//test/integration:integration_admin_test                                FAILED in 2.2s
//test/integration:integration_test                                      FAILED in 1.8s
//test/integration:legacy_json_integration_test                          FAILED in 5.5s
//test/integration:load_stats_integration_test                           FAILED in 1.6s
//test/integration:lua_integration_test                                  FAILED in 7.1s
//test/integration:metrics_service_integration_test                      FAILED in 8.8s
//test/integration:proxy_proto_integration_test                          FAILED in 2.6s
//test/integration:ratelimit_integration_test                            FAILED in 2.1s
//test/integration:squash_filter_integration_test                        FAILED in 1.8s
//test/integration:ssl_integration_test                                  FAILED in 4.8s
//test/integration:stats_integration_test                                FAILED in 1.6s
//test/integration:tcp_proxy_integration_test                            FAILED in 7.3s
//test/integration:uds_integration_test                                  FAILED in 3.1s
//test/integration:xds_integration_test                                  FAILED in 1.6s
//test/integration:xfcc_integration_test                                 FAILED in 2.8s
//test/server:listener_manager_impl_test                                 FAILED in 6.1s
//test/server:server_test                                                FAILED in 2.4s
//test/server/config_validation:server_test                              FAILED in 12.1s

[jmarantz]

More detailed example from //test/common/json:json_loader_test

[ RUN      ] JsonLoaderTest.YamlObject
/usr/include/c++/6/debug/safe_iterator.h:149:
Error: attempt to copy-construct an iterator from a singular iterator.

Objects involved in the operation:
    iterator "this" @ 0x0x7ffcc3f9f208 {
      type = __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<YAML::detail::node**, std::__cxx1998::vector<YAML::detail::node*, std::allocator<YAML::detail::node*> > >, std::__debug::vector<YAML::detail::node*, std::allocator<YAML::detail::node*> > > (mutable iterator);
      state = singular;
    }
    iterator "other" @ 0x0x7ffcc3f9f088 {
      type = __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<YAML::detail::node**, std::__cxx1998::vector<YAML::detail::node*, std::allocator<YAML::detail::node*> > >, std::__debug::vector<YAML::detail::node*, std::allocator<YAML::detail::node*> > > (mutable iterator);
      state = singular;
    }
[2018-02-08 13:43:45.172][16][critical][backtrace] bazel-out/k8-dbg/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:101] Caught Aborted, suspect faulting address 0x1109000000010
[2018-02-08 13:43:45.172][16][critical] Backtrace (most recent call first) from thread 0:

[jmarantz]

Update; I'll poke around at this a little more but there is some kind of iterator copy-construction violation in the json library which might flood the report with benign issues we have little control over.

It's still useful in a targeted stl-intensive unit test though.