Intermittent segfault in webrender build script (signal: 11, SIGSEGV: invalid memory reference)

[jdm]

Caused by:
  process didn't exit successfully: `/repo/target/debug/build/webrender-0c9654493a04f959/build-script-build` (signal: 11, SIGSEGV: invalid memory reference)
--- stdout
cargo:rerun-if-changed=res
cargo:rerun-if-changed=res/cs_gradient.glsl
cargo:rerun-if-changed=res/prim_shared.glsl
cargo:rerun-if-changed=res/pf_vector_cover.glsl
cargo:rerun-if-changed=res/pls_init.glsl
cargo:rerun-if-changed=res/cs_line_decoration.glsl
cargo:rerun-if-changed=res/composite.glsl
cargo:rerun-if-changed=res/ps_text_run.glsl
cargo:rerun-if-changed=res/gpu_cache.glsl
cargo:rerun-if-changed=res/base.glsl
cargo:rerun-if-changed=res/cs_blur.glsl
cargo:rerun-if-changed=res/brush_opacity.glsl
cargo:rerun-if-changed=res/ellipse.glsl
cargo:rerun-if-changed=res/cs_clip_image.glsl
cargo:rerun-if-changed=res/brush_mix_blend.glsl
cargo:rerun-if-changed=res/brush.glsl
cargo:rerun-if-changed=res/shared_other.glsl
cargo:rerun-if-changed=res/transform.glsl
cargo:rerun-if-changed=res/brush_conic_gradient.glsl
cargo:rerun-if-changed=res/brush_solid.glsl
cargo:rerun-if-changed=res/yuv.glsl
cargo:rerun-if-changed=res/ps_clear.glsl
cargo:rerun-if-changed=res/cs_clip_box_shadow.glsl
cargo:rerun-if-changed=res/pf_vector_stencil.glsl
cargo:rerun-if-changed=res/clip_shared.glsl
cargo:rerun-if-changed=res/cs_border_segment.glsl
cargo:rerun-if-changed=res/cs_svg_filter.glsl
cargo:rerun-if-changed=res/brush_radial_gradient.glsl
cargo:rerun-if-changed=res/cs_border_solid.glsl
cargo:rerun-if-changed=res/brush_linear_gradient.glsl
cargo:rerun-if-changed=res/brush_image.glsl
cargo:rerun-if-changed=res/ps_split_composite.glsl
cargo:rerun-if-changed=res/brush_yuv_image.glsl
cargo:rerun-if-changed=res/cs_scale.glsl
cargo:rerun-if-changed=res/rect.glsl
cargo:rerun-if-changed=res/cs_clip_rectangle.glsl
cargo:rerun-if-changed=res/pls_resolve.glsl
cargo:rerun-if-changed=res/brush_multi.glsl
cargo:rerun-if-changed=res/debug_font.glsl
cargo:rerun-if-changed=res/brush_blend.glsl
cargo:rerun-if-changed=res/debug_color.glsl
cargo:rerun-if-changed=res/render_task.glsl
cargo:rerun-if-changed=res/shared.glsl
cargo:rerun-if-changed=res/gpu_cache_update.glsl
Optimizing shader ShaderOptimizationInput { shader_name: "composite", config: "", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "composite", config: "TEXTURE_2D", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "composite", config: "TEXTURE_RECT", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "composite", config: "YUV", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "composite", config: "YUV,TEXTURE_2D", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "cs_clip_image", config: "", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "composite", config: "YUV,TEXTURE_RECT", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "brush_conic_gradient", config: "", gl_version: Gl }
Optimizing shader ShaderOptimizationInput { shader_name: "brush_conic_gradient", config: "ALPHA_PASS", gl_version: Gl }

�[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m build failed, waiting for other jobs to finish...

[jdm]

Since this happens in this code, I suspect that either glslopt is not threadsafe the way it's being used, or the implementation of build_parallel is not upholding the required Rust safety invariants.

[jdm]

aras-p/glsl-optimizer#76 suggests that glsl-optimizer might be the culprit.

[jdm]

I'm going to switch us to https://github.com/servo/webrender/tree/crash-backtrace to get a better sense of what code is segfaulting.

[jdm]

  Stack trace
     0: build_script_build::backtrace::print
               at /root/.cargo/git/checkouts/webrender-479e138e1c1b9e8b/34d968a/webrender/backtrace.rs:17
     1: build_script_build::handler
               at /root/.cargo/git/checkouts/webrender-479e138e1c1b9e8b/34d968a/webrender/build.rs:262
     2: <unknown>
     3: _ZNK9glsl_type12is_interfaceEv
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl_types.h:871
     4: _ZN11ir_variableC2EPK9glsl_typePKc16ir_variable_mode
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/ir.cpp:1833
     5: _ZN12_GLOBAL__N_126builtin_variable_generator12add_variableEPKcPK9glsl_typei16ir_variable_modei
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/builtin_variables.cpp:558
     6: _ZN12_GLOBAL__N_126builtin_variable_generator11add_uniformEPK9glsl_typeiPKc
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/builtin_variables.cpp:613
     7: _ZN12_GLOBAL__N_126builtin_variable_generator11add_uniformEPK9glsl_typePKc
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/builtin_variables.cpp:449
     8: _ZN12_GLOBAL__N_126builtin_variable_generator17generate_uniformsEv
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/builtin_variables.cpp:983
     9: _Z31_mesa_glsl_initialize_variablesP9exec_listP22_mesa_glsl_parse_state
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/builtin_variables.cpp:1553
    10: _Z16_mesa_ast_to_hirP9exec_listP22_mesa_glsl_parse_state
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/ast_to_hir.cpp:131
    11: glslopt_optimize
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/glsl-optimizer/src/compiler/glsl/glsl_optimizer.cpp:673
    12: glslopt::Context::optimize
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/glslopt-0.1.2/src/lib.rs:48
    13: build_script_build::write_optimized_shaders::{{closure}}
               at /root/.cargo/git/checkouts/webrender-479e138e1c1b9e8b/34d968a/webrender/build.rs:159
    14: core::ops::function::impls::<impl core::ops::function::Fn<A> for &F>::call
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libcore/ops/function.rs:243
    15: build_parallel::compile_object
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/build-parallel-0.1.1/src/lib.rs:24
    16: build_parallel::compile_objects::{{closure}}::{{closure}}
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/build-parallel-0.1.1/src/lib.rs:90
    17: crossbeam_utils::thread::ScopedThreadBuilder::spawn::{{closure}}
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/thread.rs:415
    18: crossbeam_utils::thread::ScopedThreadBuilder::spawn::{{closure}}
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/thread.rs:423
    19: <alloc::boxed::Box<F> as core::ops::function::FnMut<A>>::call_mut
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/liballoc/boxed.rs:1088
    20: crossbeam_utils::thread::ScopedThreadBuilder::spawn::{{closure}}
               at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/thread.rs:431
    21: std::sys_common::backtrace::__rust_begin_short_backtrace
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/sys_common/backtrace.rs:130
    22: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/thread/mod.rs:475
    23: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/panic.rs:318
    24: std::panicking::try::do_call
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/panicking.rs:348
    25: __rust_try
    26: std::panicking::try
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/panicking.rs:325
    27: std::panic::catch_unwind
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/panic.rs:394
    28: std::thread::Builder::spawn_unchecked::{{closure}}
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/thread/mod.rs:474
    29: core::ops::function::FnOnce::call_once{{vtable.shim}}
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libcore/ops/function.rs:233
    30: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/liballoc/boxed.rs:1081
        <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/liballoc/boxed.rs:1081
        std::sys::unix::thread::Thread::new::thread_start
               at /rustc/2753fab7ce3647033146b07c8b6c9f4856a910b0/src/libstd/sys/unix/thread.rs:87
    31: start_thread
    32: __clone

[jdm]

Since it's segfaulting on calling methods from a type that is created at https://github.com/jamienicol/glsl-optimizer/blob/a9bdfcc8d80050e45e40d9cf806f3e38f339afd0/src/compiler/glsl/builtin_variables.cpp#L983, I strongly suspect threading issues.

[jdm]

The various types are stored in static hashtables in https://github.com/jamienicol/glsl-optimizer/blob/a9bdfcc8d80050e45e40d9cf806f3e38f339afd0/src/compiler/glsl_types.h#L1170-L1186, which looks unlikely to be protected against multiple threads using them.

[jdm]

jamienicol/glsl-optimizer@d6ec0aa added a mutex protecting those hashtables, but I think it's being undermined by https://github.com/aras-p/glsl-optimizer/blob/21b98a985423eec771600d2db0fbf6d340106af9/src/glsl/builtin_functions.cpp#L4628-L4632.