snmp: Add ddwrt module #147

daenney · 2017-04-22T19:46:54Z

The list of SNMP OIDs to care about for DD-WRT can be found here: https://www.dd-wrt.com/wiki/index.php/SNMP#Known_OID.C2.B4s_via_SNMP

This works with routers running DD-WRT. It includes the usual things like network interfaces from SNMPv2-SMI and some information about the device itself through HOST-RESOURCES-MIB. I left out support for hrSWRun, hrSWRunPerf, hrSWInstalled and hrMIBAdminInfo as they seem bad fits for what Prometheus would care about and in the case of my device don't contain any data.

The 1.3.6.1.4.1.2021 bits come from the UCD-SNMP-MIB, part of Net-SNMP which is what appears to be the SNMPd on these devices:

.4: memory
.10: laTable (load information)
.11: systemStats
.255: unknown (used by DD-WRT to store wireless client info so I had to make up names for it)

daenney · 2017-04-22T19:48:43Z

This is the info I get from my DD-WRT from the UCDAVIS MIB, in case someone wants to check I didn't botch the types (I've 00-ified the MACs). I've mapped INTEGER to Gauge and COUNTER32 to counter, as that seems to be what the other modules do too.

UCD-SNMP-MIB::memIndex.0 = INTEGER: 0
UCD-SNMP-MIB::memErrorName.0 = STRING: swap
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 0 kB
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 0 kB
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 239368 kB
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 187012 kB
UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 187044 kB
UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB
UCD-SNMP-MIB::memShared.0 = INTEGER: 0 kB
UCD-SNMP-MIB::memBuffer.0 = INTEGER: 4544 kB
UCD-SNMP-MIB::memCached.0 = INTEGER: 24956 kB
UCD-SNMP-MIB::memSwapError.0 = INTEGER: error(1)
UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING: Running out of swap space (0)
UCD-SNMP-MIB::laIndex.1 = INTEGER: 1
UCD-SNMP-MIB::laIndex.2 = INTEGER: 2
UCD-SNMP-MIB::laIndex.3 = INTEGER: 3
UCD-SNMP-MIB::laNames.1 = STRING: Load-1
UCD-SNMP-MIB::laNames.2 = STRING: Load-5
UCD-SNMP-MIB::laNames.3 = STRING: Load-15
UCD-SNMP-MIB::laLoad.1 = STRING: 0.00
UCD-SNMP-MIB::laLoad.2 = STRING: 0.01
UCD-SNMP-MIB::laLoad.3 = STRING: 0.05
UCD-SNMP-MIB::laConfig.1 = STRING: 12.00
UCD-SNMP-MIB::laConfig.2 = STRING: 12.00
UCD-SNMP-MIB::laConfig.3 = STRING: 12.00
UCD-SNMP-MIB::laLoadInt.1 = INTEGER: 0
UCD-SNMP-MIB::laLoadInt.2 = INTEGER: 1
UCD-SNMP-MIB::laLoadInt.3 = INTEGER: 4
UCD-SNMP-MIB::laErrorFlag.1 = INTEGER: noError(0)
UCD-SNMP-MIB::laErrorFlag.2 = INTEGER: noError(0)
UCD-SNMP-MIB::laErrorFlag.3 = INTEGER: noError(0)
UCD-SNMP-MIB::laErrMessage.1 = STRING:
UCD-SNMP-MIB::laErrMessage.2 = STRING:
UCD-SNMP-MIB::laErrMessage.3 = STRING:
UCD-SNMP-MIB::ssIndex.0 = INTEGER: 1
UCD-SNMP-MIB::ssErrorName.0 = STRING: systemStats
UCD-SNMP-MIB::ssSwapIn.0 = INTEGER: 0 kB
UCD-SNMP-MIB::ssSwapOut.0 = INTEGER: 0 kB
UCD-SNMP-MIB::ssIOSent.0 = INTEGER: 0 blocks/s
UCD-SNMP-MIB::ssIOReceive.0 = INTEGER: 0 blocks/s
UCD-SNMP-MIB::ssSysInterrupts.0 = INTEGER: 477 interrupts/s
UCD-SNMP-MIB::ssSysContext.0 = INTEGER: 36 switches/s
UCD-SNMP-MIB::ssCpuUser.0 = INTEGER: 1
UCD-SNMP-MIB::ssCpuSystem.0 = INTEGER: 2
UCD-SNMP-MIB::ssCpuIdle.0 = INTEGER: 95
UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 32953
UCD-SNMP-MIB::ssCpuRawNice.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawSystem.0 = Counter32: 49885
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 1848870
UCD-SNMP-MIB::ssRawInterrupts.0 = Counter32: 9208326
UCD-SNMP-MIB::ssRawContexts.0 = Counter32: 696812
UCD-DLMOD-MIB::dlmodNextIndex.0 = INTEGER: 1
UCD-SNMP-MIB::versionIndex.0 = INTEGER: 1
UCD-SNMP-MIB::versionTag.0 = STRING: 5.0.9
UCD-SNMP-MIB::versionDate.0 = STRING: $Date: 2004/06/20 21:54:28 $
UCD-SNMP-MIB::versionCDate.0 = STRING: Sat Apr 22 21:06:51 2017
UCD-SNMP-MIB::versionIdent.0 = STRING: $Id: versioninfo.c,v 1.1.2.1 2004/06/20 21:54:28 nikki Exp $
UCD-SNMP-MIB::versionConfigureOptions.0 = STRING: \" --quiet --prefix=/tmp/snmp --target=mipsel-linux --host=mipsel '--with-cc=ccache mipsel-linux-uclibc-gcc' --with-ar=mipsel-linux-uclibc-ar --with-endianness=little '--with-cflags=-Os -pipe -mips32r2 -mtune=74kc -mdspr2 -fno-caller-saves  -msoft-float  -mno-branch-likely -minterlink-mips16 -mips16 -DCAN_USE_SYSCTL=1 -ffunction-sections -fdata-sections -Wl,--gc-sections' --enable-mini-agent --disable-debugging --disable-privacy --without-opaque-special-types --with-persistent-directory=/tmp/snmp-persist --with-default-snmp-version=3 --with-sys-contact=root --with-sys-location=Unknown --with-logfile=/dev/null --with-out-transports=UDPIPv6,TCPIPv6,AAL5PVC,IPX,TCP,Unix --enable-shared=no --enable-static --with-gnu-ld --enable-internal-md5 --with-copy-persistent-files=no --without-openssl -sysconfdir=/tmp --with-mib-modules=mibII,host,mibII/ip,mibII/tcp,mibII/udp,mibII/icmp,mibII/var_route,mibII/kernel_linux,ucd_snmp --with-out-mib-modules=host/hr_swrun,agent_mips,agentx,notification,utilities,target --disable-ip
UCD-SNMP-MIB::versionClearCache.0 = INTEGER: 0
UCD-SNMP-MIB::versionUpdateConfig.0 = INTEGER: 0
UCD-SNMP-MIB::versionRestartAgent.0 = INTEGER: 0
UCD-SNMP-MIB::versionSavePersistentData.0 = INTEGER: 0
UCD-SNMP-MIB::versionDoDebugging.0 = INTEGER: 1
UCD-SNMP-MIB::snmperrIndex.0 = INTEGER: 0
UCD-SNMP-MIB::snmperrNames.0 = STRING: snmp
UCD-SNMP-MIB::snmperrErrorFlag.0 = INTEGER: noError(0)
UCD-SNMP-MIB::snmperrErrMessage.0 = STRING:
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.1.1 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.1.2 = INTEGER: 2
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.1.3 = INTEGER: 3
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.1.4 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.1.5 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.4.1 = Hex-STRING: 00 00 00 00 00 00
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.4.2 = Hex-STRING: 00 00 00 00 00 00
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.4.3 = Hex-STRING: 00 00 00 00 00 00
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.4.4 = Hex-STRING: 00 00 00 00 00 00
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.4.5 = STRING: "hT�Q��"
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.13.1 = INTEGER: -86
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.13.2 = INTEGER: -86
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.13.3 = INTEGER: -86
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.13.4 = INTEGER: -86
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.13.5 = INTEGER: -86
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.26.1 = INTEGER: 66
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.26.2 = INTEGER: 25
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.26.3 = INTEGER: 19
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.26.4 = INTEGER: 27
UCD-SNMP-MIB::ucdavis.255.3.54.1.3.32.1.26.5 = INTEGER: 49

daenney · 2017-04-22T19:58:09Z

A general observation... there's an enormous amount of duplicate OID mangling in snmp.yml b/c a lot of devices use the same OIDs. Some do nice things like associate the xDescr with the rest of the x series as a label for easy querying, others don't. I found the syntax there to be tremendously confusing. Wouldn't it make more sense to split this out in one module per MIB, so have snmpv2smi, hostresources, ucdsnmp and only have ddwrt walk the extra OIDs it cares about, just as to similarly limit apcups to just walking 1.3.6.1.4.1.318? Or maybe something more generic like network, system and then vendor specifics like apcups?

Since modules appears to be an array it should be possible to have a target be checked by multiple modules?

daenney · 2017-04-22T22:28:36Z

I'm running into a very odd problem. I think it's a bug in DD-WRT honestly but I'm wondering if there's some way to work around it.

It seems that there are hrStorageIndex entries that have the exact same hrStorageDescr entry:

# HELP hrStorageIndex 
# TYPE hrStorageIndex gauge
hrStorageIndex{hrStorageIndex="1"} 1
[..]
hrStorageIndex{hrStorageIndex="2"} 2
[..]

# HELP hrStorageDescr 
# TYPE hrStorageDescr gauge
hrStorageDescr{hrStorageDescr="/",hrStorageIndex="1"} 1
hrStorageDescr{hrStorageDescr="/",hrStorageIndex="2"} 1
[..]

This makes it impossible to do the lookups trick to then be able to associate hrStorageDescr as label with the rest of the hrStorage* entries since the exporter will rightfully complain that

collected metric hrStorageDescr label:<name:"hrStorageDescr" value:"/" > gauge:<value:1 >  was collected before with the same name and label values

Is it possible to work around that? I tried keeping two indexes on the entry but that just results in an empty label being added to it?

SuperQ · 2017-04-22T22:33:46Z

These need to be added to the generator configuration.

SuperQ · 2017-04-22T22:38:28Z

You're right about not duplicating the default walk. And only implement the specific walks for the non-default network metrics. Breaking things up by system in more generic modules like you suggest is probably a good idea, since a lot of these are standardized MIBs, and not ddwrt specific.

Of course, it'd be way better to use the node_exporter with a ddwrt box, since you can skip SNMP altogether. 😄

Another ddwrt option would be to use collectd, since it has the option to provide a prometheus target endpoint.

daenney · 2017-04-22T22:42:54Z

I spent two days trying to get node_exporter to run on my router :P. Switched firmware a few times over, Merlin one's kernel was too old but it turns out DD-WRT compiles for MIPS without FPU emulation which makes Go break. It's been a long long day 😛. Hence why I went down this road.

SuperQ · 2017-04-22T22:45:23Z

Yea, embedded systems can be a huge pain.

daenney · 2017-04-22T22:59:39Z

Eh, so how do I add the 1.3.6.1.4.1.2021.255 to generator? The .255 is unknown so is basically used to stash other things in but as a consequence there's no proper names for those OIDs that it can piggy-back on since no MIB exists for it.

In that same spirit, do I need to do something specific once I've amended generator, like run it and include the result in the PR too?

brian-brazil · 2017-04-23T08:36:02Z

there's no proper names for those OIDs that it can piggy-back on since no MIB exists for it.

We need the MIB, as that's how SNMP is designed. If one doesn't already exist, your best option in this case is probably to contribute one to dd-wrt and make sure it ends up somewhere easy to download.

In that same spirit, do I need to do something specific once I've amended generator, like run it and include the result in the PR too?

Yes. You'll also need to include a pointer to the MIB in the README.

daenney · 2017-04-23T09:53:59Z

We need the MIB, as that's how SNMP is designed. If one doesn't already exist, your best option in this case is probably to contribute one to dd-wrt and make sure it ends up somewhere easy to download.

Hmm, might be a while, if ever, for them to pull that in. But I can create one and host it on a GitHub repository for now. Would that be enough for the scope of this PR?

daenney · 2017-04-23T10:17:32Z

Slightly different question. Lets say for hrStorageCapacity it currently looks like hrDiskStorageCapacity{hrDeviceIndex="1552"} 3.0253056e+07 (basically 32GB). Is there any way to make it get both hrDeviceIndex and hrDeviceDescr as labels on that thing? Based on the lookup I could get hrDeviceIndex replaced by hrDeviceDescr, but what if I wanted both (for programatic reason)?

SuperQ · 2017-04-23T10:56:57Z

@daenney If you created a valid MIB, that's good enough for me.

For the labels, officially you should only include label lookups that are minimally unique. So you would have to pick one of the two.

In practice, I do this all the time0.

daenney · 2017-04-23T12:12:13Z

Alright, in that case I'll amend the PR to include the defined entries from UCDAVIS and deal with the .255 range separately as it might take me some time to whip up a MIB for that. It's been years... I'll maintain an artisanally handcrafted snmp.yml somewhere else with the added bits and extra labels for now.

Final question, I think. There are some entries from laTable, 1.3.6.1.4.1.2021.10, that are really not worth scraping/including. Is there any way to indicate that? laLoad for example is useless in this context since it's a DisplayString of a load, like 0.01. However since that value changes all the time you end up with a humongous amount of time series which all have laLoad{laLoad=string} 1 as a result which is just bonkers since we have laLoadInt that you can just divide by 100 or laLoadFloat for systems that get that.

SuperQ · 2017-04-23T12:29:43Z

There's no current option in the generator.yml to allow for an ignored OID list. This would be a useful feature.

For now, you have to be a bit verbose in the walk list and only include the exact list you want included.

daenney · 2017-04-23T13:42:50Z

I'm struggling a bit to get the snmp-mibs-downloader to work. I added cisco.conf with:

HOST=ftp://ftp.cisco.com
ARCHIVE=v2.tar.gz
ARCHTYPE=tgz
DIR=pub/mibs/v2/
ARCHDIR=auto/mibs/v2
CONF=ciscolist
DEST=cisco

And a ciscolist that basically mentions every MIB in the archive (though it probably shouldn't?):

ACCOUNTING-CONTROL-MIB.my
ACTONA-ACTASTOR-MIB.my
ADMIN-AUTH-STATS-MIB.my
ADSL-DMT-LINE-MIB.my
ADSL-LINE-MIB.my
[..]
T11-TC-MIB.my
TCP-MIB.my
TN3270E-MIB.my
TN3270E-RT-MIB.my
TOKENRING-MIB.my
TUNNEL-MIB.my
UDP-MIB.my
VDSL2-LINE-MIB.my
VDSL2-LINE-TC-MIB.my
VPN-TC-STD-MIB.my
VRRP-MIB.my
XGCP-MIB.my
draft-ietf-ipfc-fcmgmt-int-mib-04.my
test-mib.my

However, running a download-mibs cisco just results in a whole bunch of this:

NOTE: CISCO-ATM-ACCESS-LIST-MIB: ignored.
WARNING: Module(s) not found: -
NOTE: CISCO-ATM-ADDR-MIB: ignored.
WARNING: Module(s) not found: -
NOTE: CISCO-ATM-CAPABILITY: ignored.

daenney · 2017-04-23T13:54:50Z

Oh, it's a dual column format, file name + MIB name. My bad.

daenney · 2017-04-23T14:08:39Z

Alright. I think that's it. Thank you so much for your help and guidance @SuperQ, @brian-brazil!

brian-brazil · 2017-04-23T14:14:42Z

But I can create one and host it on a GitHub repository for now. Would that be enough for the scope of this PR?

We would prefer the MIB from the vendor, otherwise we can't know that you got all the details right.

daenney · 2017-04-23T14:21:41Z

We would prefer the MIB from the vendor, otherwise we can't know that you got all the details right.

There is no MIB from the vendor unfortunately. Someone contributed a simple script to DD-WRT named wl_snmpd.sh that's just hooking into that part of the tree. I can try and contribute one to them but it's largely based on goodwill from their side to even keep that up to date / in sync. For now I've left it out of this PR.

daenney · 2017-04-23T14:26:58Z

I've removed hrSystem, it's largely useless on this device. The only interesting one was uptime that sysUpTime already provides.

brian-brazil · 2017-04-24T10:09:45Z

generator/generator.yml

+      - interfaces
+      - ifXTable
+      - 1.3.6.1.2.1.25.2 # hrStorage
+      - 1.3.6.1.4.1.2021.4 # ucdavis mem


this object is called memory

brian-brazil · 2017-04-24T10:11:40Z

generator/generator.yml

+      - 1.3.6.1.2.1.25.2 # hrStorage
+      - 1.3.6.1.4.1.2021.4 # ucdavis mem
+      # ucdavis laTable (load avg)
+      - 1.3.6.1.4.1.2021.10.1.1 # laIndex


This is called processIndex in the MIB I'm looking at. Are you sure you're using the right MIB?

Are you sure you're looking at the right MIB: http://www.oidview.com/mibs/2021/UCD-SNMP-MIB.html? prIndex is at 1.3.6.1.4.1.2021.2.1.1

Ah indeed, I was looking at the wrong object.

brian-brazil · 2017-04-24T13:07:46Z

generator/generator.yml

+      - 1.3.6.1.4.1.2021.10.1.1 # laIndex
+      - 1.3.6.1.4.1.2021.10.1.2 # laNames
+      - 1.3.6.1.4.1.2021.10.1.5 # laLoadInt
+      - 1.3.6.1.4.1.2021.11 # ucdavis systemStats


just systemStats

Why? The other one where I'm requesting everything is prefixed with ucdavis, like ucdavis memory. The laLoad one has a header of ucdavis laLoad and then the three OIDs that are of interest explicitly listed. Consistency wise, this is what's done before. I'm happy to change it but then I'd like to do this once so if you can tell me what you'd like the whole block to look like I'm happy to amend it.

The convention in this file is for the comment to be just the object name, without any reference to the module name.

Alright. Done. I've updated all the comments to reflect that then.

The list of SNMP OIDs to care about for DD-WRT can be found here: https://www.dd-wrt.com/wiki/index.php/SNMP#Known_OID.C2.B4s_via_SNMP This works with routers running DD-WRT. It includes the usual things like network interfaces from SNMPv2-SMI and some information about the device itself through HOST-RESOURCES-MIB. I left out support for `hrSWRun`, `hrSWRunPerf`, `hrSWInstalled` and `hrMIBAdminInfo` as they seem bad fits for what Prometheus would care about and in the case of my device don't contain any data. Same thing for `hrDevice`. The `1.3.6.1.4.1.2021` bits come from the UCD-SNMP-MIB, part of Net-SNMP which is what appears to be the SNMPd on these devices: * .10: laTable (load information) * .11: systemStats

brian-brazil · 2017-04-24T13:32:20Z

Thanks!

daenney · 2017-04-24T13:33:27Z

Thank you too!

* [FEATURE] Add Homepage on /. prometheus#135 * [IMPROVEMENT] Add ddwrt OIDs to generator. prometheus#147 * [IMPROVEMENT] Add synology OIDs to generator. prometheus#149, prometheus#154 * [IMPROVEMENT] Use lookup node's index label in the generator. prometheus#162 * [BUGFIX] Fix `authNoPriv` in config parsing. prometheus#141 * [BUGFIX] Update gosnmp vendoring to fix timeouts/errors. prometheus#139, prometheus#171

brian-brazil reviewed Apr 24, 2017

View reviewed changes

brian-brazil merged commit 5da2d9c into prometheus:master Apr 24, 2017

daenney deleted the ddwrt branch April 24, 2017 13:33

SuperQ mentioned this pull request Jun 6, 2017

Release v0.4.0 #172

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

snmp: Add ddwrt module #147

snmp: Add ddwrt module #147

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 22, 2017

SuperQ commented Apr 22, 2017

SuperQ commented Apr 22, 2017

daenney commented Apr 22, 2017

SuperQ commented Apr 22, 2017

daenney commented Apr 22, 2017 •

edited

Loading

brian-brazil commented Apr 23, 2017

daenney commented Apr 23, 2017

daenney commented Apr 23, 2017 •

edited

Loading

SuperQ commented Apr 23, 2017 •

edited

Loading

daenney commented Apr 23, 2017 •

edited

Loading

SuperQ commented Apr 23, 2017

daenney commented Apr 23, 2017

daenney commented Apr 23, 2017

daenney commented Apr 23, 2017

brian-brazil commented Apr 23, 2017

daenney commented Apr 23, 2017 •

edited

Loading

daenney commented Apr 23, 2017

brian-brazil Apr 24, 2017

daenney Apr 24, 2017

brian-brazil Apr 24, 2017

daenney Apr 24, 2017 •

edited

Loading

brian-brazil Apr 24, 2017

brian-brazil Apr 24, 2017

daenney Apr 24, 2017

brian-brazil Apr 24, 2017

daenney Apr 24, 2017

brian-brazil commented Apr 24, 2017

daenney commented Apr 24, 2017

snmp: Add ddwrt module #147

snmp: Add ddwrt module #147

Conversation

daenney commented Apr 22, 2017 • edited Loading

daenney commented Apr 22, 2017 • edited Loading

daenney commented Apr 22, 2017 • edited Loading

daenney commented Apr 22, 2017

SuperQ commented Apr 22, 2017

SuperQ commented Apr 22, 2017

daenney commented Apr 22, 2017

SuperQ commented Apr 22, 2017

daenney commented Apr 22, 2017 • edited Loading

brian-brazil commented Apr 23, 2017

daenney commented Apr 23, 2017

daenney commented Apr 23, 2017 • edited Loading

SuperQ commented Apr 23, 2017 • edited Loading

daenney commented Apr 23, 2017 • edited Loading

SuperQ commented Apr 23, 2017

daenney commented Apr 23, 2017

daenney commented Apr 23, 2017

daenney commented Apr 23, 2017

brian-brazil commented Apr 23, 2017

daenney commented Apr 23, 2017 • edited Loading

daenney commented Apr 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daenney Apr 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brian-brazil commented Apr 24, 2017

daenney commented Apr 24, 2017

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 22, 2017 •

edited

Loading

daenney commented Apr 23, 2017 •

edited

Loading

SuperQ commented Apr 23, 2017 •

edited

Loading

daenney commented Apr 23, 2017 •

edited

Loading

daenney commented Apr 23, 2017 •

edited

Loading

daenney Apr 24, 2017 •

edited

Loading