r15 beta2 openMP #401

rcketscientist · 2017-05-21T18:51:19Z

Description

I am getting errors while building with clang and openMP. Note that clang is clearly throwing the error, but it seems to be referencing gcc 4.9 versions of the library. Could I possibly have something in my (ndk-build) make files goofing this up?

D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_get_wtick has undefined version OMP_2.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_get_default_device has undefined version OMP_4.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_destroy_lock has undefined version VERSION
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_get_active_level has undefined version OMP_3.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_destroy_nest_lock has undefined version OMP_1.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_in_final has undefined version OMP_3.1
clang++.exe: error: linker command failed with exit code 1 (use -v to see invocation)

Environment Details

NDK Version: 15.0.3996722-beta2
Build sytem: ndk-build
Host OS: Windows
Compiler: Clang before filing.
ABI: arm v7
STL: gnustl_static

donlk · 2017-05-22T10:23:28Z

Same linkage error for me during OpenCV build, though i noticed instead of parsing the ususal 'fopemp' flag, 'fopenmp=libomp' gets used.

DanAlbert · 2017-05-22T18:21:28Z

Could I possibly have something in my (ndk-build) make files goofing this up?

Could you share your Android.mk/Application.mk?

rcketscientist · 2017-05-22T19:29:40Z

Application.mk

#APP_OPTIM := release moved to gradle

# openmp with clang, unsupported till r14 or r15
APP_LDFLAGS += -fopenmp -Wl,--no-undefined

ADDITIONAL_FLAGS := -fopenmp

APP_CFLAGS += $(ADDITIONAL_FLAGS)

APP_CPPFLAGS += $(ADDITIONAL_FLAGS) -std=c++11

APP_LDLIBS += -lz

APP_PLATFORM := android-15

#armeabi is deprecated in 4.4, this is handled in build.gradle now
#APP_ABI := armeabi-v7a arm64-v8a

APP_STL := gnustl_static

#build broken in ndk 13.0.3315539 with default clang
#NDK_TOOLCHAIN_VERSION := 4.9

Android.mk (openMP project)

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE := libraw

#path to LibRAW source root:
LIBRAW_PATH := $(LOCAL_PATH)/LibRaw

FILE_LIST := \
	swab.c \
	$(LIBRAW_PATH)/internal/dcraw_common.cpp \
	$(LIBRAW_PATH)/internal/dcraw_fileio.cpp \
	$(LIBRAW_PATH)/internal/demosaic_packs.cpp \
	$(LIBRAW_PATH)/src/libraw_datastream.cpp \
	$(LIBRAW_PATH)/src/libraw_cxx.cpp
	
LOCAL_SRC_FILES := $(FILE_LIST)
LOCAL_C_INCLUDES := $(LIBRAW_PATH)/internal $(LIBRAW_PATH)/libraw $(LIBRAW_PATH)/		
LOCAL_EXPORT_C_INCLUDES	:= $(LIBRAW_PATH)/libraw

# Warning flags ==================================================

LOCAL_CFLAGS := -w
#LOCAL_CFLAGS += -Wno-deprecated-declarations #suppress auto_ptr warnings (doesn't work)
LOCAL_CFLAGS += -Wno-c++11-narrowing # suppress libraw_x3f narrowing warnings

# Optional =======================================================

#enable threading
LOCAL_CFLAGS += -pthread

# CMS
LOCAL_CFLAGS += -DUSE_LCMS2
LOCAL_STATIC_LIBRARIES += lcms2

# JPEG
LOCAL_CFLAGS += -DUSE_JPEG
LOCAL_STATIC_LIBRARIES += jpeg-turbo

# DNG
# LOCAL_CFLAGS += -DUSE_DNGSDK
# LOCAL_STATIC_LIBRARIES += dng

# Demosaic Pack GPL2 (only for foveon back compat):
#LOCAL_CFLAGS += -DLIBRAW_DEMOSAIC_PACK_GPL2
#LOCAL_C_INCLUDES += $(LOCAL_PATH)/LibRaw-demosaic-pack-GPL2

# Demosaic Pack GPL3 (AMaZE, not sure use-case):
#LOCAL_CFLAGS += -DLIBRAW_DEMOSAIC_PACK_GPL3
#LOCAL_C_INCLUDES += $(LOCAL_PATH)/LibRaw-demosaic-pack-GPL3

#==================================================================
#pthread sets some rentrant flags that will fully enable threading
LOCAL_CPPFLAGS := $(LOCAL_CFLAGS) -fexceptions -frtti

include $(BUILD_STATIC_LIBRARY)

DanAlbert · 2017-05-22T19:43:26Z

What about the shared library that uses the static library? Is that part of the same project? You need to pass -fopenmp for every module that uses libraw, so if you're pulling thins library into some other project that doesn't have that flag, that's the problem.

rcketscientist · 2017-05-22T19:55:18Z

gcc worked without passing for every module if by module you mean every ndk sub-project. All ndk projects are built within the same imageprocessor library below under the Application.mk linked earlier.

Android.mk (parent)

LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)

LOCAL_MODULE    := imageprocessor
  
LOCAL_LDLIBS    += -llog

LOCAL_SRC_FILES := 	src/imageprocessor.cpp src/libraw_descriptor_stream.cpp

#disable fatal warning: shared library text segment is not shareable
LOCAL_DISABLE_FATAL_LINKER_WARNINGS := true

LOCAL_STATIC_LIBRARIES := jpeg-turbo raw tiff #exif
#LOCAL_SHARED_LIBRARIES := dng

include $(BUILD_SHARED_LIBRARY)

DanAlbert · 2017-05-22T20:00:29Z

The parent Android.mk uses the same Application.mk then? If you run ndk-build V=1, do you see -fopenmp being passed when libimageprocessor is being linked?

rcketscientist · 2017-05-23T15:16:27Z

D:/proj/sdk/ndk-bundle/build//../toolchains/llvm/prebuilt/windows-x86_64/bin/clang++.exe -Wl,-soname,libimageprocessor.so -shared --sysroot=D:/proj/sdk/ndk-bundle/build//../platforms/android-15/arch-arm D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/objs/imageprocessor/src/imageprocessor.o D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/objs/imageprocessor/src/libraw_descriptor_stream.o D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/libraw.a D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/libtiff.a D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/liblcms2.a D:/proj/sdk/ndk-bundle/build//../sources/cxx-stl/gnu-libstdc++/4.9/libs/armeabi-v7a/libgnustl_static.a D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/libjpeg-turbo.a -lgcc -Wl,--exclude-libs,libgcc.a -gcc-toolchain D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64 -no-canonical-prefixes -target armv7-none-linux-androideabi15 -Wl,--fix-cortex-a8 -Wl,--build-id -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--warn-shared-textrel -fopenmp -Wl,--no-undefined -LD:/proj/sdk/ndk-bundle/build//../platforms/android-15/arch-arm/usr/lib -llog -lz -lc -lm -o D:/proj/rawdroid/rawprocessor/obj/local/armeabi-v7a/libimageprocessor.so
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_get_wtick has undefined version OMP_2.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_get_default_device has undefined version OMP_4.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_destroy_lock has undefined version VERSION
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_get_active_level has undefined version OMP_3.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_destroy_nest_lock has undefined version OMP_1.0
D:/proj/sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: symbol omp_in_final has undefined version OMP_3.1
clang++.exe: error: linker command failed with exit code 1 (use -v to see invocation)

DanAlbert · 2017-05-23T20:25:39Z

Thanks for confirming that.

Hmm. I'm still not able to reproduce this locally. Any chance one of the libraries you're linking against uses openmp and was built with GCC? The two are probably not compatible, but can't tell from looking which if any of your dependencies might have that problem. Regarding your concern in your initial report, that it might be using the GCC 4.9 library, the section you've highlighted is actually just showing that it's using binutils, which happens to be in the GCC directory. Nothing so far stands out as incorrect.

I don't suppose your project is open source? I'd have an easier time tracking things down if I can reproduce the failure myself.

donlk · 2017-05-23T20:34:33Z

I strongly advise you to try and build the most recent stable release of OpenCV with r15-beta2 armv7a clang standalone toolchain, using their own android.toolchain.cmake descriptor. I get the exact same link errors, and i absolutely link against no prebuilt libraries, so ABI collision is unlikely (I still could be wrong, if the system tries to force the linkage against some exotic system library, but i've yet to see that happen on arm cmake builds with OpenCV).

DanAlbert · 2017-05-23T20:43:28Z

Latest stable release being 3.2.0? I don't get very far at all with that.

$ ANDROID_NDK=~/src/android-ndk-r15-beta2 ~/src/CMake/bin/cmake -DCMAKE_TOOLCHAIN_FILE=../platforms/android/android.toolchain.cmake ..
CMake Deprecation Warning at /work/src/CMake/Modules/CMakeForceCompiler.cmake:69 (message):
  The CMAKE_FORCE_C_COMPILER macro is deprecated.  Instead just set
  CMAKE_C_COMPILER and allow CMake to identify the compiler.
Call Stack (most recent call first):
  platforms/android/android.toolchain.cmake:1154 (CMAKE_FORCE_C_COMPILER)
  /work/src/CMake/Modules/CMakeDetermineSystem.cmake:91 (include)
  CMakeLists.txt:98 (project)


CMake Deprecation Warning at /work/src/CMake/Modules/CMakeForceCompiler.cmake:83 (message):
  The CMAKE_FORCE_CXX_COMPILER macro is deprecated.  Instead just set
  CMAKE_CXX_COMPILER and allow CMake to identify the compiler.
Call Stack (most recent call first):
  platforms/android/android.toolchain.cmake:1166 (CMAKE_FORCE_CXX_COMPILER)
  /work/src/CMake/Modules/CMakeDetermineSystem.cmake:91 (include)
  CMakeLists.txt:98 (project)


-- Detected version of GNU GCC: 49 (409)
-- Looking for ccache - found (/usr/bin/ccache)
CMake Error at cmake/OpenCVCompilerOptions.cmake:21 (else):
  A duplicate ELSE command was found inside an IF block.
Call Stack (most recent call first):
  CMakeLists.txt:495 (include)


-- Configuring incomplete, errors occurred!
See also "/work/src/opencv/out/CMakeFiles/CMakeOutput.log".

Build instructions would really speed things up here.

donlk · 2017-05-23T20:57:34Z

If the android toolchain file sees your ANDROID_NDK variable, it will prioratize it over the standalone choice, and you get whatever is set as default, currently GCC. I only use standalone toolchains for this exact reason, i cannot comment on the behavior you get with the NDK.

Also, make sure you pass this flag as well:
-DANDROID_TOOLCHAIN_NAME=standalone-clang

DanAlbert · 2017-05-23T21:19:40Z

That doesn't change anything (though I did try, just in case). The error is one in the opencv cmake files themselves:

CMake Error at cmake/OpenCVCompilerOptions.cmake:21 (else):
  A duplicate ELSE command was found inside an IF block.
Call Stack (most recent call first):
  CMakeLists.txt:495 (include)

What version of opencv? What options for the standalone toolchain?

Could you please provide concrete repro instructions?

donlk · 2017-05-23T21:43:25Z

Its OpenCV 3.2.0 as you sad. Try this link: https://github.com/opencv/opencv/archive/3.2.0.tar.gz

So i sat down and reproduced the issue with the following steps:

1. Extract a standalone toolchain from your NDK:
<NDK>/build/tools/make_standalone_toolchain.py --arch arm --api 21 --deprecated-headers --install-dir <INSTALL_DIR>
Api level makes no difference here, but make sure you use the deprecated-headers, as the unified ones are still buggy with cmake.
2. Feed the the following flags to cmake from an empty build directory of your choice inside the OpenCV source tree:
cmake -DCMAKE_TOOLCHAIN_FILE=<OPENCV>/platforms/android/android.toolchain.cmake -DANDROID_TOOLCHAIN_NAME=standalone-clang -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DCMAKE_C_FLAGS="-fopenmp" -DCMAKE_CXX_FLAGS="-fopenmp" -DCMAKE_SHARED_LINKER_FLAGS="-fopenmp" ../

And you end up with something similar:
[ 36%] Linking CXX shared library ../../lib/armeabi-v7a/libopencv_core.so SDK/Android/toolchain/r15-beta2-api-21-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: symbol omp_get_wtick has undefined version OMP_2.0 SDK/Android/toolchain/r15-beta2-api-21-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: symbol omp_get_cancellation has undefined version OMP_4.0 SDK/Android/toolchain/r15-beta2-api-21-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: symbol omp_destroy_nest_lock has undefined version OMP_3.0 SDK/Android/toolchain/r15-beta2-api-21-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: symbol omp_destroy_lock has undefined version OMP_1.0 SDK/Android/toolchain/r15-beta2-api-21-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: symbol omp_destroy_nest_lock has undefined version VERSION SDK/Android/toolchain/r15-beta2-api-21-toolchain/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: symbol omp_in_final has undefined version OMP_3.1 clang50: error: linker command failed with exit code 1 (use -v to see invocation)

EDIT: I forgot to say: BUILD_SHARED_LIBS=ON is key here. It seems to build fine if you build in static library mode. Which makes me wonder if this is the root of the issue, as libomp is only present in static form in the NDK, so probably it gets ignored for shared libraries(?).

DanAlbert · 2017-05-23T21:55:59Z

Ah. So the error with the duplicate else only shows up with newer versions of cmake. I was able to repro the issue now. Thanks for the instructions!

but make sure you use the deprecated-headers as the unified ones are still buggy with cmake.

Do you have a bug for this? I didn't have any issues using unified headers for this project. We don't know of any issues other than #302 and #394 (really the same issue). Based on your phrasing, it sounds like you know of something cmake specific? If you don't file them, I can't fix them :)

donlk · 2017-05-23T22:20:03Z

I get the weirdest bugs with unified headers and cmake.
For example i bump into these errors at OpenCV configure step:
CMake Error at platforms/android/android.toolchain.cmake:315 (message): Could not get Android native API level. Probably you have specified invalid level value, or your copy of NDK/toolchain is broken. Call Stack (most recent call first): platforms/android/android.toolchain.cmake:508 (__DETECT_NATIVE_API_LEVEL) /usr/share/cmake-3.7/Modules/CMakeDetermineSystem.cmake:88 (include) CMakeLists.txt:98 (project)

This is probably due to the value 10000 for ANDROID_API in api-level.h, as it is an unstable NDK release. But even if i set it to something like __ANDROID_API__ 21, this one comes up:

-- FP16: Compiler support is not available -- Check if the system is big endian -- Searching 16 bit integer CMake Error at /usr/share/cmake-3.7/Modules/TestBigEndian.cmake:41 (message): no suitable type found Call Stack (most recent call first): CMakeLists.txt:548 (test_big_endian)

Only after explicitly passing LITTLE_ENDIAN does it work.
It seems like some of the basic flags are not getting passed automatically somehow. It might be the fault of OpenCV's android.toolchain.cmake descriptor, i'm not sure, but after you guys excluded standalone toolchain support from NDK's own toolchain descriptor, i cannot use that one.

DanAlbert · 2017-05-23T22:37:41Z

Ah. So the opencv cmake toolchain just doesn't support them (a good reason to use ours, if that's possible). With a standalone toolchain that's not a problem though (I used one to repro this without issue). Both of those issues are symptoms of their toolchain file not using unified headers correctly.

Anyway, back to the real problem. @pirama-arumuga-nainar: any idea how the symbol versioning in libomp is supposed to work for a static library? The version script never comes in to play when we build it because there's no link step. Do we need to just disable symbol versioning for the static library? I see there's a KMP_USE_VERSION_SYMBOLS config that controls this. It looks like that gets set as part of the configure step.

I think whether or not this is the right long term solution, we should do this for r15 unless someone has a better idea.

One other thing that would be good would be to find a minimized test case for this. None of the openmp tests we have in the NDK trip this. I've extended the openmp tests to be built with CMake as well, but that didn't trip this (I didn't expect it to).

rcketscientist · 2017-05-24T07:45:45Z

All libraries are built from source in my case. The project is closed source, but I have no problem giving you access to debug this. The project using openMP is open source, LibRaw, but it requires some modification for Android. If the openCV test case isn't enough let me know. I'm making an international move tomorrow, so I'm uncertain when or if I'll have time to help here for a while.

DanAlbert · 2017-05-24T07:56:58Z

OpenCV was enough to repro the issue. Thanks though!

DanAlbert · 2017-05-30T23:02:38Z

https://android-review.googlesource.com/c/405044/ is the fix. Was able to build opencv using your instructions with that change. Will have the fix in r15 stable. Thanks for the help tracking this down!

DanAlbert · 2017-06-01T23:12:31Z

New compiler is merged into r15.

pirama-arumuga-nainar · 2017-12-22T18:52:43Z

I discovered that this accidentally regressed in r16. We changed how Clang is built and packaged and OpenMP's version symbols got turned on by mistake. However a proper fix will be included the next time NDK Clang gets updated (r17 maybe?).

In the mean time, I found that the following version script avoids the link failures (at the cost of exporting all symbols from libomp.a):

VERSION {
};

# sets up GCC OMP_ version dependency chain
OMP_1.0 {
};
OMP_2.0 {
} OMP_1.0;
OMP_3.0 {
} OMP_2.0;
OMP_3.1 {
} OMP_3.0;
OMP_4.0 {
} OMP_3.1;

# sets up GCC GOMP_ version dependency chain
GOMP_1.0 {
};
GOMP_2.0 {
} GOMP_1.0;
GOMP_3.0 {
} GOMP_2.0;
GOMP_4.0 {
} GOMP_3.0;

Test: ./run_tests.py --filter test-openmp Bug: android/ndk#401 Change-Id: I855d8a1a4c1a333f0cebd2ad2fd142073bca48f5

Test: ./checkbuild.py && ./run_tests.py Bug: android/ndk#401 Change-Id: Ie548c38bb8a20a9c748712c8332882031014485f

DanAlbert self-assigned this May 23, 2017

DanAlbert added this to the r15 milestone May 23, 2017

DanAlbert mentioned this issue May 30, 2017

Clang not compiled with OpenMP support #9

Closed

DanAlbert closed this as completed Jun 1, 2017

miodragdinic pushed a commit to MIPS/ndk that referenced this issue Apr 17, 2018

Build the openmp test with CMake too.

64d0586

Test: ./run_tests.py --filter test-openmp Bug: android/ndk#401 Change-Id: I855d8a1a4c1a333f0cebd2ad2fd142073bca48f5

miodragdinic pushed a commit to MIPS/ndk that referenced this issue Apr 17, 2018

Update to clang-4053586.

2ae16a8

Test: ./checkbuild.py && ./run_tests.py Bug: android/ndk#401 Change-Id: Ie548c38bb8a20a9c748712c8332882031014485f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

r15 beta2 openMP #401

r15 beta2 openMP #401

rcketscientist commented May 21, 2017 •

edited

donlk commented May 22, 2017 •

edited

DanAlbert commented May 22, 2017

rcketscientist commented May 22, 2017

DanAlbert commented May 22, 2017

rcketscientist commented May 22, 2017

DanAlbert commented May 22, 2017

rcketscientist commented May 23, 2017

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 •

edited

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 •

edited

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 •

edited

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 •

edited

DanAlbert commented May 23, 2017

rcketscientist commented May 24, 2017

DanAlbert commented May 24, 2017

DanAlbert commented May 30, 2017

DanAlbert commented Jun 1, 2017

pirama-arumuga-nainar commented Dec 22, 2017

r15 beta2 openMP #401

r15 beta2 openMP #401

Comments

rcketscientist commented May 21, 2017 • edited

Description

Environment Details

donlk commented May 22, 2017 • edited

DanAlbert commented May 22, 2017

rcketscientist commented May 22, 2017

Application.mk

Android.mk (openMP project)

DanAlbert commented May 22, 2017

rcketscientist commented May 22, 2017

Android.mk (parent)

DanAlbert commented May 22, 2017

rcketscientist commented May 23, 2017

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 • edited

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 • edited

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 • edited

DanAlbert commented May 23, 2017

donlk commented May 23, 2017 • edited

DanAlbert commented May 23, 2017

rcketscientist commented May 24, 2017

DanAlbert commented May 24, 2017

DanAlbert commented May 30, 2017

DanAlbert commented Jun 1, 2017

pirama-arumuga-nainar commented Dec 22, 2017

rcketscientist commented May 21, 2017 •

edited

donlk commented May 22, 2017 •

edited

donlk commented May 23, 2017 •

edited

donlk commented May 23, 2017 •

edited

donlk commented May 23, 2017 •

edited

donlk commented May 23, 2017 •

edited