F103RE variant problems #719

holla2040 · 2019-11-26T21:43:15Z

Folks,
I'm writing after 8 or so hours of tinkering with F103FB and F103RE variants. I have a custom board with STM32F103RET6. Using blink.ino and STLINK (no maple bootloader), using the STM32F103RB variant, works fine, when I switch to STM32F103RE, blink doesn't start. In fact, after I flash the RE bin, the CPU no longer responds to st-info --probe. I then have to use openocd to unlock the processor.

I've checked the linker scripts to make sure the ram and rom locations and lengths are correct. Verified this in the map file.

The RE variant defines, MCU_STM32F103RE, which in turn defines
define STM32_SRAM_END ((void*)0x20010000)
define STM32_HIGH_DENSITY

STM32_SRAM_END is correct for 64K ram.

defining STM32_HIGH_DENSITY, affects these files.

cores/maple/HardwareTimer.cpp
cores/maple/HardwareTimer.h
cores/maple/libmaple/adc_f1.c
cores/maple/libmaple/dma_f1.c
cores/maple/libmaple/rcc_f1.c
cores/maple/libmaple/spi_f1.c
cores/maple/libmaple/stm32f1/performance/isrs.S
cores/maple/libmaple/stm32f1/performance/vector_table.S
cores/maple/libmaple/timer.c
cores/maple/libmaple/usart_f1.c
cores/maple/sdio.cpp
cores/maple/tone.cpp
libraries/MapleCoOS116/examples/coos_display_blink/SPICoOS.cpp
libraries/SPI/src/SPI.cpp
system/libmaple/include/libmaple/bkp.h
system/libmaple/include/libmaple/sdio.h
system/libmaple/stm32f1/include/series/adc.h
system/libmaple/stm32f1/include/series/dma.h
system/libmaple/stm32f1/include/series/spi.h
system/libmaple/stm32f1/include/series/stm32.h
system/libmaple/stm32f1/include/series/timer.h
system/libmaple/stm32f1/include/series/usart.h
system/support/doxygen/Doxyfile

I'm now in the process of looking through these files to figure out my problem. Any advice on where to start?

Thanks,
Craig

stevstrong · 2019-12-19T17:22:56Z

I think the RE variant is not complete, or is not error free.
Unfortunately, I have no such board so I cannot help on this.

victorpv · 2019-12-19T20:25:22Z

I have compiled for RC and RE board successfully on older versions of the core (used one of those to write the very first version of the stm32f1 marlin HAL and ran fine). It's possible that something broke since then. The easiest way would be to hook up a debugger and see exactly where it's crashing.
Lacking a debugging probe, you may need to insert either led blinks or serial prints in different parts of the code. Test first just blinking in setup() before even opening the serial or usb port, so you can see if the problem is with initialization within the core, or is it later when running your code or trying to use the peripherals. If the blink there doesn't work, start going back somewhere between the start of main() and the call to setup().

holla2040 · 2019-12-19T20:32:03Z

Thanks for the comments. At some point, I'll use a debugger to find where it hangs.

victorpv · 2019-12-20T16:05:32Z

Have you tried the RC MCU option to see if that works or not?

stevstrong · 2019-12-26T08:19:31Z

Any update?

holla2040 · 2019-12-26T20:09:56Z

Not yet, taking this week off, need a quiet break. Got it jotted down for Monday 12/30.

holla2040 · 2019-12-26T21:32:09Z

OK, I didn't wait until Monday.

I'm compiling on Ubunut 16.04 using arduino-builder command line (taken from arduino 1.8.10 verbose output). Compiling with -fqbn=Arduino_STM32:STM32F1:genericSTM32F103R:device_variant=STM32F103RE,upload_method=STLinkMethod,cpu_speed=speed_72mhz,opt=osstd

in start_c.c, function start_c, line 84,
83 /* Run initializers. */
84 __libc_init_array();

the call to __libc_init_array never returns.

variant STM32F103RB returns from __libc_init_array just fine.

So I'll investigate more.

victorpv · 2019-12-26T23:11:33Z

That's some progress, I believe that takes care on initializing memory for object instances. Most likely failing to initialize a peripheral that's different betweenthe RB andRE versions. Can you try the same using RC or RD as MCU (RD possibly not available as an option, but RC I believe should be). I remember at least the DMA peripheral is different, and the UARTs, SPI, I2S, and a few more if I remember right are different too.
Do you have anything other thank blinking in the sketch, such as opening a serial port or anything else?
I remember now I used an STM32F103RET6 to write the I2S driver a couple of years ago, and adapted an MP3 player to run on it too, so back then everything was working fine. I should have some boards with that MCU, will try to blink a led if I have a moment. I would have to download the latest core.

victorpv · 2019-12-26T23:12:52Z

BTW if you have one of those super cheap ST-Link clones, it can be turned into a j-link, and then use Ozone to debug which is awesome.

holla2040 · 2019-12-27T00:23:00Z

Not at my desk at the moment... I'll see if the RC variant hangs in the same spot. No serial in blink sketch, just I/O. I'll check the build output, I think there's a -DSERIAL_USB flag or something like that. Thanks for replying.

…

On Thu, Dec 26, 2019, 4:11 PM victorpv ***@***.***> wrote: That's some progress, I believe that takes care on initializing memory for object instances. Most likely failing to initialize a peripheral that's different betweenthe RB andRE versions. Can you try the same using RC or RD as MCU (RD possibly not available as an option, but RC I believe should be). I remember at least the DMA peripheral is different, and the UARTs, SPI, I2S, and a few more if I remember right are different too. Do you have anything other thank blinking in the sketch, such as opening a serial port or anything else? I remember now I used an STM32F103RET6 to write the I2S driver a couple of years ago, and adapted an MP3 player to run on it too, so back then everything was working fine. I should have some boards with that MCU, will try to blink a led if I have a moment. I would have to download the latest core. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#719?email_source=notifications&email_token=AAH2EHSN4ZLDHDCLO2JEU2DQ2U22NA5CNFSM4JR6GQT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHWIEIA#issuecomment-569147936>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH2EHT7BBJ3AKZVULADZHTQ2U22NANCNFSM4JR6GQTQ> .

holla2040 · 2019-12-27T01:16:35Z

OK RC variant, hangs on __libc_init_array().

Here's the build output for start_s.c and syscalls.c ( -DSERIAL_USB is a GCC define, don't know if that helps or not).

/mnt/ramdisk/arduino/arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gcc -c -g -x assembler-with-cpp -MMD -mcpu=cortex-m3 -DF_CPU=72000000L -DARDUINO=10810 -DARDUINO_GENERIC_STM32F103R -DARDUINO_ARCH_STM32F1 -DCONFIG_MAPLE_MINI_NO_DISABLE_DEBUG=1 -DSERIAL_USB -DGENERIC_BOOTLOADER -DMCU_STM32F103RC -mthumb -march=armv7-m -D__STM32F1__ -DMCU_STM32F103RC -mthumb -march=armv7-m -D__STM32F1__ -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/include -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/stm32f1/include -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/usb/stm32f1 -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/usb/usb_lib -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/cores/maple -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/variants/generic_stm32f103r /mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/variants/generic_stm32f103r/wirish/start.S -o /tmp/arduino_build/core/wirish/start.S.o
===info ||| Progress {0} ||| [143.75]
===info ||| Progress {0} ||| [143.75]
/mnt/ramdisk/arduino/arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gcc -c -g -Os -w -DDEBUG_LEVEL=DEBUG_NONE -std=gnu11 -MMD -ffunction-sections -fdata-sections -nostdlib --param max-inline-insns-single=500 -DBOARD_generic_stm32f103r -DVECT_TAB_ADDR=0x8000000 -DERROR_LED_PORT=GPIOB -DERROR_LED_PIN=1 -mcpu=cortex-m3 -DF_CPU=72000000L -DARDUINO=10810 -DARDUINO_GENERIC_STM32F103R -DARDUINO_ARCH_STM32F1 -DCONFIG_MAPLE_MINI_NO_DISABLE_DEBUG=1 -DSERIAL_USB -DGENERIC_BOOTLOADER -DMCU_STM32F103RC -mthumb -march=armv7-m -D__STM32F1__ -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/include -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/stm32f1/include -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/usb/stm32f1 -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/usb/usb_lib -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/cores/maple -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/variants/generic_stm32f103r /mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/variants/generic_stm32f103r/wirish/start_c.c -o /tmp/arduino_build/core/wirish/start_c.c.o
/mnt/ramdisk/arduino/arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gcc -c -g -Os -w -DDEBUG_LEVEL=DEBUG_NONE -std=gnu11 -MMD -ffunction-sections -fdata-sections -nostdlib --param max-inline-insns-single=500 -DBOARD_generic_stm32f103r -DVECT_TAB_ADDR=0x8000000 -DERROR_LED_PORT=GPIOB -DERROR_LED_PIN=1 -mcpu=cortex-m3 -DF_CPU=72000000L -DARDUINO=10810 -DARDUINO_GENERIC_STM32F103R -DARDUINO_ARCH_STM32F1 -DCONFIG_MAPLE_MINI_NO_DISABLE_DEBUG=1 -DSERIAL_USB -DGENERIC_BOOTLOADER -DMCU_STM32F103RC -mthumb -march=armv7-m -D__STM32F1__ -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/include -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/stm32f1/include -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/usb/stm32f1 -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/system/libmaple/usb/usb_lib -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/cores/maple -I/mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/variants/generic_stm32f103r /mnt/ramdisk/arduino/Arduino/hardware/Arduino_STM32/STM32F1/variants/generic_stm32f103r/wirish/syscalls.c -o /tmp/arduino_build/core/wirish/syscalls.c.o

Now, after it hangs on __libc_init_array, st-link can't find the MCU anymore.

holla@hendrix[1275]: st-info --probe
Found 1 stlink programmers
 serial: 513f6e067267504817360267
openocd: "\x51\x3f\x6e\x06\x72\x67\x50\x48\x17\x36\x02\x67"
  flash: 0 (pagesize: 0)
   sram: 0
 chipid: 0x0000
  descr: unknown device

I have to unlock the MCU with
openocd -f interface/stlink.cfg -f target/stm32f1x.cfg -c "init;reset halt;stm32f1x unlock 0;reset halt;exit"

I did find this link https://community.arm.com/developer/ip-products/processors/f/cortex-m-forum/11187/memory-related-issue-with-mcu-startup-__libc_init_array

About half way down, there's a couple comments. Here's my attempts at workaround. -falign-functions=4 doesn't fix this problem. As for the optimization, I haven't figured out how arduino-builder sets the optimization level yet.

victorpv · 2019-12-27T02:29:46Z

Seems like you are having a hard fault. I did confirm I wrote the i2s and sdio drivers for that core on STM32F103RET6 boards. From the date of the folder in arduino/hardware, it was a version from around 04/02/2018 or earlier. I likely modified multiple other things in that core to use a 16Mhz crystal since that's what I used in the bluepill F4 boards I fitted with F103RET MCUs.

I'm downloading the current repo and will try to recompile one of those projects with it.
I use Sloeber/Eclipse normally.

One more thing, what optimization level did you use? O0, O2, O3, LTO or not LTO?
I didn't have much luck with LTO back then, but looks like the mp3 player code was compiled with O2 LTO and 128Mhz pll settings. Can't remember if that was working or was just me trying to squeeze a bit more of performance, but those were the settings the last time it was compiled last year.

One more thing, I remember dealing with hard crashes a long time ago similar to what you get. I ended up running thru the initialization with ozone one function at a time, and it was crashing when initializing an SPI port instance. That was back when I was doing changes to the SPI library to support DMA or improve the DMA functions a bit more. That's why I suspect yours is crashing on the creator of an instance of some peripheral.

Anyway if I can get something to crash in the new place with the new core, I may be able to help.

victorpv · 2019-12-27T02:57:31Z

Ok, this was fast. I downloaded the current core, zip file from github, uncompressed, and changed my SDIO project to use that core.
Set the MCU to STM32F103RE, use stm32duino boot loader, and recompile.
The MCU in this chinese board is an STM32F103RCT6 but I have confirmed before it had 512/64 memory like an RET, so I think is a relabeled RET6 MCU.
The code got recompiled (confirmed because the bin got deleted when I changed core, and the new .bin size is a different size also).
Loaded the code to the board, which previously had a beta of Marlin, so I know the bin got loaded to the board since the serial output was for Marlin before, and now is for my SDIO test sketch.
The program runs fine, doesn't find the SD card cause there is none connected, but it starts, initializes, start writing to the serial USB port, and reports it couldn't access the SD card. Here is the output:

######################################################
  Demo sketch of STM32F1 SDIO (DMA) implementation.
######################################################
SdFatSdio begin() failed
Can't access SD card. Do not reformat.
SD errorCode: 0X25,0X0

At this point I can't reproduce the problem even with a more complex sketch, so I think you should try downloading the core again just in case there was a bug in the version you have in your hard drive.
As compile optimizations I chose fastest(default), and not-LTO.

Didn't try blinking a led, but I can't imagine it would get this far and fail to blink a led.

holla2040 · 2019-12-27T03:37:47Z

OK, sounds good. I'll change the optimization and see what happens. Thanks for spending the time on this, I appreciate it.

victorpv · 2019-12-27T04:39:39Z

Remember to download the core again, since I tested with the version from today and you opened the issue last month, so perhaps something got fixed in between.

holla2040 · 2019-12-27T04:51:21Z

Sure, I cloned the repo this afternoon. I'll work a little more this evening, first trying the -O3 change.

holla2040 · 2019-12-27T06:32:55Z

No luck with RE with any optimization.

blink.ino

const int LED_BUILTIN=PC13;

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop() {
  delay(100);
  digitalWrite(LED_BUILTIN, LOW);
  delay(100);
  digitalWrite(LED_BUILTIN, HIGH);
}

I'm compiling using linux IDE 1.8.10 only, all libraries and boards are up to date (no command line compiles anymore). Tried both cloned and zip download of Arduino_STM32 this evening. Using STLINK/v2 cheap clone for programmer.

RB compiles and runs with all optimizations. RE compiles and does not run with any optimization.

I'm going to try these same steps on Windows 10 now.

rogerclarkmelbourne · 2019-12-27T06:37:44Z

Some of the core and the libraries will only work OK if you use the default optimisation of -OS

AFIK -Os is basically -O2 except for things which would make the code bigger.

I think the core will work OK with -O3 but not all the libraries work OK with that level of optimisation.

IMHO its not worth using the LTO settings as the code hardly seems any smaller or faster with any of the LTO options selected and often causes the core to break.

holla2040 · 2019-12-27T08:30:30Z

I'm not using LTO, command line opt to arduino-builder is o3std

-fqbn=Arduino_STM32:STM32F1:genericSTM32F103R:device_variant=STM32F103RE,upload_method=STLinkMethod,cpu_speed=speed_72mhz,opt=o3std

Win10 attempt
Same result with fresh install of IDE 1.8.10 and Arduino_STM32, blink.ino above for RB small and O3 work fine, RC and RE small or O3 don't.

holla2040 · 2019-12-27T09:44:39Z

OK, I think I'm on to something. I stepped through init and found the hang in setup_adcs()

in adc_f1.c adc_foreach, I commented out
// fn(ADC3);

void adc_foreach(void (*fn)(adc_dev*)) {
    fn(ADC1);
    fn(ADC2);
#if defined(STM32_HIGH_DENSITY) || defined(STM32_XL_DENSITY)
//    fn(ADC3);
#endif
}

and RE O3 started running.

Its almost 3AM in Colorado. Going to bed, I'll continue debugging tomorrow.

holla2040 · 2019-12-27T10:13:00Z

checked one more thing, RE ADC3 setup dies in adc_enable

static inline void adc_enable(adc_dev *dev) {
  *bb_perip(&dev->regs->CR2, ADC_CR2_ADON_BIT) = 1;
}

OK, now I'm going to bed.

holla2040 · 2019-12-27T17:41:17Z

This morning I tried to figure out why adc_enable(ADC3) fails

*bb_perip(&dev->regs->CR2, ADC_CR2_ADON_BIT) = 1;

fails. ADC3 dev address is correct. ADC3->CR1 is 0x00, so EOCIE is not set.

Seeking advice what to look for. Thanks.

stevstrong · 2019-12-28T09:41:25Z

I think the problem is that there is no entry for RE chip in this list:
https://github.com/rogerclarkmelbourne/Arduino_STM32/blob/master/STM32F1/system/libmaple/stm32f1/include/series/stm32.h#L108-L130
You can try to copy the settings from RB variant and paste it as RE.

victorpv · 2019-12-28T14:24:40Z

There is, in line 126.
It's weird it's crashing when seeing the adcs because I compiled for the RE with the latest code and the init completed right. If I have a chance I'll try later to write a sketch that reads from adc.
Did you check the ID of the MCU in the board? I'm wondering if it's a relabeled one, it is just faulty...

holla2040 · 2019-12-28T16:25:58Z

Hadn't thought to check the ID, good idea. Mine is listed as F1 high-density with 64K/512K which is 103RE. You might be on to something with the relabeling.

holla@hendrix[511]: st-info --probe
Found 1 stlink programmers
serial: 513f6e067267504817360267
openocd: "\x51\x3f\x6e\x06\x72\x67\x50\x48\x17\x36\x02\x67"
flash: 524288 (pagesize: 2048)
sram: 65536
chipid: 0x0414
descr: F1 High-density device

holla2040 · 2019-12-28T16:49:28Z

I verified my part has 512K flash by writing and reading a random file to flash.

victorpv · 2019-12-31T04:50:31Z

So the ID is matching for an RE MCU. I am honestly running out of ideas. Do you have a second board to test?

holla2040 · 2019-12-31T06:27:01Z

Not yet, I'm doing an initial prototype verification. Then I'll spin the board for first version. Please leave this issue open for a while and I'll get back to you. I may have something in about 3/4 weeks. Or maybe I should just switch the MCU on this board, see check results. That can happen almost immediately. I'll do that next week. Thanks for verifying and the comments on this issue. Happy New Year and hope you have a bright 2020!

…

On Mon, Dec 30, 2019 at 9:50 PM victorpv ***@***.***> wrote: So the ID is matching for an RE MCU. I am honestly running out of ideas. Do you have a second board to test? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#719?email_source=notifications&email_token=AAH2EHXKSMYDJV5NP6XV6UTQ3LFRPA5CNFSM4JR6GQT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH3W2NY#issuecomment-569863479>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH2EHRE2J6O4LSLSA2RH7TQ3LFRPANCNFSM4JR6GQTQ> .

-- Dr. Craig Hollabaugh craig@hollabaugh.com https://www.linkedin.com/in/craighollabaugh/ 970 690 4911

victorpv · 2019-12-31T08:07:58Z

Can you upload your bin somewhere? I can test flashing it to my board to confirm if it does the same.

holla2040 · 2019-12-31T14:14:06Z

Here's my non-functional blink build directory, ie fn(ADC3) is compiled in.

http://hollabaugh.com/arduino/blink.build.zip

victorpv · 2019-12-31T16:09:45Z

which pin should be blinking if it works fine?

holla2040 · 2019-12-31T16:41:29Z

PC13 and PA0

holla2040 · 2019-12-31T17:43:52Z

Also, when I run this bin file, the leds are both on, running dim and my cpu becomes locked. stlink can't read its ID. I have to unlock it to be usable again.

holla2040 · 2019-12-31T20:08:47Z

I noticed something else that might point to bad part. My previously tested EEPROM flash code doesn't work correctly either. EEPROM.read always returns 0xFF. Let me switch out the MCU on this board and get back to you. Thanks.

holla2040 · 2020-01-10T22:30:33Z

I swapped the MCU today with another 103RET from a different lot and different supplier (Digikey vs LCSC). Same result with my blink script. MCU hangs on ADC3 initialization and MCU has to be unlocked to be flashable again.

stevstrong · 2020-01-12T08:17:48Z

checked one more thing, RE ADC3 setup dies in adc_enable
static inline void adc_enable(adc_dev *dev) {
  *bb_perip(&dev->regs->CR2, ADC_CR2_ADON_BIT) = 1;
}

Could you eventually debug line by line in instruction mode so that you can check the registers and stack pointer at the exact location where the MCU hangs.

holla2040 · 2020-01-13T05:47:04Z

will do.

holla2040 · 2020-01-19T18:00:10Z

Well, I'm frustrated now. After 3 hours of trying to find where enabling ADC3 dies and get inconsistent results. Probably because I'm not very experienced with gdb. I'm compiling with device_variant=STM32F103RE and opt=ogstd (debugging).

void init(void) {
    setup_flash();
    setup_clocks();
    setup_nvic();
    systick_init(SYSTICK_RELOAD_VAL);
    wirish::priv::board_setup_gpio();
    setup_adcs();
    setup_timers();
    wirish::priv::board_setup_usb();
    wirish::priv::series_init();
    boardInit();
}

setup_adcs eventually calls adc_set_sample_rate, I can single step through
adc_set_sample_rate() my blink code then blink dies in init call to setup_timers. So I reset my board, this time if I put a breakpoint on init - setup_timers() and continue, then my blink code dies in setup_adcs().

I'm not able to find the exact spot where f103re blink fails. Again, this probably due to my gdb inexperience.

stevstrong · 2021-01-24T10:36:16Z

You have to step on assembly lines, not C code lines.

stevstrong assigned holla2040 Dec 26, 2019

stevstrong added this to the Jan, 10 milestone Dec 26, 2019

F103RE variant problems #719

F103RE variant problems #719

Comments

holla2040 commented Nov 26, 2019

stevstrong commented Dec 19, 2019

victorpv commented Dec 19, 2019

holla2040 commented Dec 19, 2019

victorpv commented Dec 20, 2019

stevstrong commented Dec 26, 2019

holla2040 commented Dec 26, 2019 • edited

holla2040 commented Dec 26, 2019

victorpv commented Dec 26, 2019

victorpv commented Dec 26, 2019 • edited

holla2040 commented Dec 27, 2019 via email

holla2040 commented Dec 27, 2019 • edited by stevstrong

victorpv commented Dec 27, 2019

victorpv commented Dec 27, 2019 • edited by stevstrong

holla2040 commented Dec 27, 2019

victorpv commented Dec 27, 2019

holla2040 commented Dec 27, 2019

holla2040 commented Dec 27, 2019

rogerclarkmelbourne commented Dec 27, 2019

holla2040 commented Dec 27, 2019

holla2040 commented Dec 27, 2019

holla2040 commented Dec 27, 2019

holla2040 commented Dec 27, 2019

stevstrong commented Dec 28, 2019

victorpv commented Dec 28, 2019 • edited

holla2040 commented Dec 28, 2019

holla2040 commented Dec 28, 2019

victorpv commented Dec 31, 2019

holla2040 commented Dec 31, 2019 via email

victorpv commented Dec 31, 2019

holla2040 commented Dec 31, 2019

victorpv commented Dec 31, 2019

holla2040 commented Dec 31, 2019 via email • edited

holla2040 commented Dec 31, 2019

holla2040 commented Dec 31, 2019

holla2040 commented Jan 10, 2020

stevstrong commented Jan 12, 2020 • edited

holla2040 commented Jan 13, 2020

holla2040 commented Jan 19, 2020

stevstrong commented Jan 24, 2021

holla2040 commented Dec 26, 2019 •

edited

victorpv commented Dec 26, 2019 •

edited

holla2040 commented Dec 27, 2019 •

edited by stevstrong

victorpv commented Dec 27, 2019 •

edited by stevstrong

victorpv commented Dec 28, 2019 •

edited

holla2040 commented Dec 31, 2019 via email •

edited

stevstrong commented Jan 12, 2020 •

edited