Storable build hangs when building 5.27.9 on Windows XP #16428

p5pRT · 2018-02-21T18:46:33Z

Migrated from rt.perl.org#132893 (status was 'resolved')

Searchable as RT132893$

p5pRT · 2018-02-21T18:46:33Z

From @chorny

Created by @chorny

..\perl.exe -I..\lib -I. ..\dist\Storable\stacksize --core
probe for max. stack sizes...
65000 failed, try less 32550 ...
32550 failed, try less 16325 ...
16325 failed, try less 8213 ...
8213 failed, try less 4157 ...
4157 passed, try more 6185 ...
6185 passed, try more 7199 ...
7199 failed, try less 6692 ...
6692 failed, try less 6439 ...
6439 failed, try less 6312 ...
6312 failed, try less 6249 ...
6249 failed, try less 6217 ...
6217 failed, try less 6201 ...
6201 failed, try less 6193 ...
6193 passed, try more 6197 ...
6197 passed, try more 6199 ...
6199 passed, try more 6200 ...
6200 passed, try more 6200 ...
MAX_DEPTH = 6200
3100 passed, try more 4650 ...
4650 passed, try more 5425 ...
5425 passed, try more 5812 ...
Out of memory!
(hangs)

Also `stacksize` is executed on every stage - build/tests/install, so
I had to kill it 3 times. After killing hanging process, stacksize
continues search.
This is same problem as with Storable on CPAN. In CPAN version I
solved it with Win32::Job, but it is not a core module. See
rurban/Storable#2
It should be possible to use Win32::Process, but it is not a core module either.

cpantesters results show that with RURBAN/Storable-3.05_16.tar.gz
these values can be partially predicted. Note that values for 3.05_16
on CPAN and 3.06 in core are different. And `stacksize` in 3.05_16
currently does not hang even when not using Win32::Job.

Windows 10 64-bit, perl 5.26.0 64-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.22.1 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.22.1 64-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.24.0 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.20.1 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.18.2 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.18.2 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=512
Windows XP 32-bit inside VM, perl 5.26.0 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=512
Windows XP 32-bit inside VM, perl 5.12.2 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit, perl 5.27.9 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit, perl 5.14.0 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.16.0, 5.16.3 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257

(Total 2 installations of Windows 10 and 3 installations of Windows XP)

http://www.cpantesters.org/cpan/report/11bf30ab-6bf4-1014-b5fc-ed671f639c1c
Windows 7 ?-bit (probably 32-bit), perl 5.14.4 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257

Perl Info


Flags:
    category=library
    severity=high
    module=Storable

Site configuration information for perl 5.27.9:

Configured by strawberry-perl at Wed Feb 21 04:27:05 2018.

Summary of my perl5 (revision 5 version 27 subversion 9) configuration:

  Platform:
    osname=MSWin32
    osvers=5.1.2600
    archname=MSWin32-x86-multi-thread-64int
    uname='Win32 strawberry-perl 5.27.9.1-beta1 #1 Wed Feb 21 04:24:03
2018 i386'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -s -O2 -DWIN32 -D__USE_MINGW_ANSI_STDIO
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='7.1.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=4
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=12
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++.exe'
    ldflags ='-s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'
    libpth=C:\strawberry\c\lib C:\strawberry\c\i686-w64-mingw32\lib
C:\strawberry\c\lib\gcc\i686-w64-mingw32\7.1.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl527.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE" -L"C:\strawberry\c\lib"'



@INC for perl 5.27.9:
    C:/strawberry/perl/site/lib
    C:/strawberry/perl/vendor/lib
    C:/strawberry/perl/lib


Environment for perl 5.27.9:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\strawberry\c\bin;C:\Program
Files\Far\;C:\Prg\PHP\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program
Files\ATI Technologies\ATI.ACE\Core-Static;C:\strawberry514\c\bin;C:\strawberry514\perl\site\bin;C:\strawberry514\perl\bin;C:\Program
Files\TortoiseHg\;C:\Program Files\MySQL\MySQL Server
5.5\bin;c:\util\;C:\Prg\Subversion\bin;C:\Program
Files\TortoiseSVN\bin;D:\vagrant\vagrant\bin;C:\Prg\TortoiseGit\bin;C:\Prg\Git\cmd;C:\Program
Files\Skype\Phone\;
    PERL_BADLANG (unset)
    SHELL (unset)


-- 
Alexandr Ciornii, http://chorny.net

p5pRT · 2018-02-21T19:00:38Z

From @jkeenan

On Wed, 21 Feb 2018 18:46:33 GMT, chorny wrote:

This is a bug report for perl from alexchorny@gmail.com,
generated with the help of perlbug 1.41 running under perl 5.27.9.

-----------------------------------------------------------------
[Please describe your issue here]

..\perl.exe -I..\lib -I. ..\dist\Storable\stacksize --core
probe for max. stack sizes...
65000 failed, try less 32550 ...
32550 failed, try less 16325 ...
16325 failed, try less 8213 ...
8213 failed, try less 4157 ...
4157 passed, try more 6185 ...
6185 passed, try more 7199 ...
7199 failed, try less 6692 ...
6692 failed, try less 6439 ...
6439 failed, try less 6312 ...
6312 failed, try less 6249 ...
6249 failed, try less 6217 ...
6217 failed, try less 6201 ...
6201 failed, try less 6193 ...
6193 passed, try more 6197 ...
6197 passed, try more 6199 ...
6199 passed, try more 6200 ...
6200 passed, try more 6200 ...
MAX_DEPTH = 6200
3100 passed, try more 4650 ...
4650 passed, try more 5425 ...
5425 passed, try more 5812 ...
Out of memory!
(hangs)

Also `stacksize` is executed on every stage - build/tests/install, so
I had to kill it 3 times. After killing hanging process, stacksize
continues search.
This is same problem as with Storable on CPAN. In CPAN version I
solved it with Win32::Job, but it is not a core module. See
rurban/Storable#2
It should be possible to use Win32::Process, but it is not a core
module either.

cpantesters results show that with RURBAN/Storable-3.05_16.tar.gz

Note that that's an unauthorized release: http://search.cpan.org/~rurban/Storable-3.05_16/

these values can be partially predicted. Note that values for 3.05_16
on CPAN and 3.06 in core are different. And `stacksize` in 3.05_16
currently does not hang even when not using Win32::Job.

Windows 10 64-bit, perl 5.26.0 64-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.22.1 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.22.1 64-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.24.0 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.20.1 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.18.2 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.18.2 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=512
Windows XP 32-bit inside VM, perl 5.26.0 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=512
Windows XP 32-bit inside VM, perl 5.12.2 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit, perl 5.27.9 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit, perl 5.14.0 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.16.0, 5.16.3 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257

(Total 2 installations of Windows 10 and 3 installations of Windows
XP)

http://www.cpantesters.org/cpan/report/11bf30ab-6bf4-1014-b5fc-
ed671f639c1c
Windows 7 ?-bit (probably 32-bit), perl 5.14.4 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=library
severity=high
module=Storable
---
Site configuration information for perl 5.27.9:

Configured by strawberry-perl at Wed Feb 21 04:27:05 2018.

Summary of my perl5 (revision 5 version 27 subversion 9)
configuration:

Platform:
osname=MSWin32
osvers=5.1.2600
archname=MSWin32-x86-multi-thread-64int
uname='Win32 strawberry-perl 5.27.9.1-beta1 #1 Wed Feb 21 04:24:03
2018 i386'
config_args='undef'
hint=recommended
useposix=true
d_sigaction=undef
useithreads=define
usemultiplicity=define
use64bitint=define
use64bitall=undef
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='gcc'
ccflags =' -s -O2 -DWIN32 -D__USE_MINGW_ANSI_STDIO
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields'
optimize='-s -O2'
cppflags='-DWIN32'
ccversion=''
gccversion='7.1.0'
gccosandvers=''
intsize=4
longsize=4
ptrsize=4
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=12
longdblkind=3
ivtype='long long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='long long'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='g++.exe'
ldflags ='-s -L"C:\strawberry\perl\lib\CORE"
-L"C:\strawberry\c\lib"'
libpth=C:\strawberry\c\lib C:\strawberry\c\i686-w64-mingw32\lib
C:\strawberry\c\lib\gcc\i686-w64-mingw32\7.1.0
libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=
so=dll
useshrplib=true
libperl=libperl527.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs
dlext=xs.dll
d_dlsymun=undef
ccdlflags=' '
cccdlflags=' '
lddlflags='-mdll -s -L"C:\strawberry\perl\lib\CORE"
-L"C:\strawberry\c\lib"'

---
@INC for perl 5.27.9:
C:/strawberry/perl/site/lib
C:/strawberry/perl/vendor/lib
C:/strawberry/perl/lib

---
Environment for perl 5.27.9:
HOME (unset)
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\strawberry\c\bin;C:\Program
Files\Far\;C:\Prg\PHP\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program
Files\ATI Technologies\ATI.ACE\Core-
Static;C:\strawberry514\c\bin;C:\strawberry514\perl\site\bin;C:\strawberry514\perl\bin;C:\Program
Files\TortoiseHg\;C:\Program Files\MySQL\MySQL Server
5.5\bin;c:\util\;C:\Prg\Subversion\bin;C:\Program
Files\TortoiseSVN\bin;D:\vagrant\vagrant\bin;C:\Prg\TortoiseGit\bin;C:\Prg\Git\cmd;C:\Program
Files\Skype\Phone\;
PERL_BADLANG (unset)
SHELL (unset)

--
James E Keenan (jkeenan@cpan.org)

p5pRT · 2018-02-21T19:00:38Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2018-02-22T04:25:48Z

From @tonycoz

On Wed, 21 Feb 2018 10:46:33 -0800, chorny wrote:

..\perl.exe -I..\lib -I. ..\dist\Storable\stacksize --core
probe for max. stack sizes...
65000 failed, try less 32550 ...
32550 failed, try less 16325 ...
16325 failed, try less 8213 ...
8213 failed, try less 4157 ...
4157 passed, try more 6185 ...
6185 passed, try more 7199 ...
7199 failed, try less 6692 ...
6692 failed, try less 6439 ...
6439 failed, try less 6312 ...
6312 failed, try less 6249 ...
6249 failed, try less 6217 ...
6217 failed, try less 6201 ...
6201 failed, try less 6193 ...
6193 passed, try more 6197 ...
6197 passed, try more 6199 ...
6199 passed, try more 6200 ...
6200 passed, try more 6200 ...
MAX_DEPTH = 6200
3100 passed, try more 4650 ...
4650 passed, try more 5425 ...
5425 passed, try more 5812 ...
Out of memory!
(hangs)

I've reproduced this. It requires USE_64_BIT_INT, I haven't managed to track down the cause yet.

Also `stacksize` is executed on every stage - build/tests/install, so
I had to kill it 3 times. After killing hanging process, stacksize
continues search.
This is same problem as with Storable on CPAN. In CPAN version I
solved it with Win32::Job, but it is not a core module. See
rurban/Storable#2
It should be possible to use Win32::Process, but it is not a core
module either.

I'll see if I can prevent running the probe several times, probably by having the rule in win32/(GNU)?[mM]akefile(.mk)? run a sub-make in the Storable directory instead of running it directly.

cpantesters results show that with RURBAN/Storable-3.05_16.tar.gz
these values can be partially predicted. Note that values for 3.05_16
on CPAN and 3.06 in core are different. And `stacksize` in 3.05_16
currently does not hang even when not using Win32::Job.

Windows 10 64-bit, perl 5.26.0 64-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.22.1 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.22.1 64-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows 10 64-bit, perl 5.24.0 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.20.1 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.18.2 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.18.2 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=512
Windows XP 32-bit inside VM, perl 5.26.0 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=512
Windows XP 32-bit inside VM, perl 5.12.2 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit, perl 5.27.9 32-bit+64-bit int:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit, perl 5.14.0 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257
Windows XP 32-bit inside VM, perl 5.16.0, 5.16.3 32-bit:
PST_STACK_MAX_DEPTH=513
PST_STACK_MAX_DEPTH_HASH=257

Those values are just stacksize falling back to a default.

A predefined table is going to vary based on bitness of build, debug/non-debug, whether USE_64_BIT_INT is set, version of perl, compiler, compiler version and changes to Storable itself.

The builder of perl may adjust the stack size too.

That's just too many dimensions.

I may just do what Reini did and fallback to a default on difficult platforms like XP.

Tony

p5pRT · 2018-02-27T23:20:46Z

From @tonycoz

On Wed, 21 Feb 2018 20:25:48 -0800, tonyc wrote:

I'll see if I can prevent running the probe several times, probably by
having the rule in win32/(GNU)?[mM]akefile(.mk)? run a sub-make in the
Storable directory instead of running it directly.

Done in b58c2b0.

I'll look at the hang/memory issue next.

Tony

p5pRT · 2018-02-28T23:23:26Z

From @tonycoz

On Tue, 27 Feb 2018 15:20:46 -0800, tonyc wrote:

On Wed, 21 Feb 2018 20:25:48 -0800, tonyc wrote:

I'll see if I can prevent running the probe several times, probably by
having the rule in win32/(GNU)?[mM]akefile(.mk)? run a sub-make in the
Storable directory instead of running it directly.

Done in b58c2b0.

I'll look at the hang/memory issue next.

XP and Vista builds now skip the probe.

Tony

p5pRT · 2019-08-06T05:14:19Z

From @tonycoz

On Wed, 28 Feb 2018 15:23:26 -0800, tonyc wrote:

On Tue, 27 Feb 2018 15:20:46 -0800, tonyc wrote:

On Wed, 21 Feb 2018 20:25:48 -0800, tonyc wrote:

I'll see if I can prevent running the probe several times, probably by
having the rule in win32/(GNU)?[mM]akefile(.mk)? run a sub-make in the
Storable directory instead of running it directly.

Done in b58c2b0.

I'll look at the hang/memory issue next.

XP and Vista builds now skip the probe.

All builds now skip the probe, it ended up being too flaky.

This was included in perl 5.30.

Thanks for the report.

Tony

p5pRT · 2019-08-06T05:14:19Z

@tonycoz - Status changed from 'open' to 'resolved'

p5pRT closed this Aug 6, 2019

p5pRT added Severity High affects-5.27 type-library labels Oct 19, 2019

Perl/perl5

Storable build hangs when building 5.27.9 on Windows XP #16428

Storable build hangs when building 5.27.9 on Windows XP #16428

p5pRT commented Feb 21, 2018

This comment has been minimized.

p5pRT commented Feb 21, 2018

This comment has been minimized.

p5pRT commented Feb 21, 2018

This comment has been minimized.

p5pRT commented Feb 21, 2018

This comment has been minimized.

p5pRT commented Feb 22, 2018

This comment has been minimized.

p5pRT commented Feb 27, 2018

This comment has been minimized.

p5pRT commented Feb 28, 2018

This comment has been minimized.

p5pRT commented Aug 6, 2019

This comment has been minimized.

p5pRT commented Aug 6, 2019

Perl/perl5

Join GitHub today

Storable build hangs when building 5.27.9 on Windows XP #16428

Comments

p5pRT commented Feb 21, 2018

This comment has been minimized.

p5pRT commented Feb 21, 2018

From @chorny

Created by @chorny

This comment has been minimized.

p5pRT commented Feb 21, 2018

From @jkeenan

This comment has been minimized.

p5pRT commented Feb 21, 2018

This comment has been minimized.

p5pRT commented Feb 22, 2018

From @tonycoz

This comment has been minimized.

p5pRT commented Feb 27, 2018

From @tonycoz

This comment has been minimized.

p5pRT commented Feb 28, 2018

From @tonycoz

This comment has been minimized.

p5pRT commented Aug 6, 2019

From @tonycoz

This comment has been minimized.

p5pRT commented Aug 6, 2019