ModemManager: MTU not being set based on Bearer info #11383

jonesmz · 2020-02-19T00:39:10Z

Maintainer: @nickberry17
Description:

When Netifd brings a ModemManager based connection up, the bearer informs Netifd of the MTU that the connection supports.

For example, an AT&T cellphone connection frequently has an MTU of below 1500 (e.g. https://www.digi.com/support/knowledge-base/recommended-mtu-mru-settings-on-cellular-networks)

For example:

root@:~# mmcli -b 31
  --------------------------------
  General            |  dbus path: /org/freedesktop/ModemManager1/Bearer/31
                     |       type: default
  --------------------------------
  Status             |  connected: yes
                     |  suspended: no
                     |  interface: wwan0
                     | ip timeout: 20
  --------------------------------
  Properties         |    roaming: allowed
  --------------------------------
  IPv4 configuration |     method: static
                     |    address: 10.52.41.111
                     |     prefix: 27
                     |    gateway: 10.52.41.112
                     |        dns: 172.26.38.1, 10.0.0.252
                     |        mtu: 1430

However, netifd is not using that information to set the MTU of the wwan0 interface that it brings up.

This results in some traffic getting silently dropped.

The text was updated successfully, but these errors were encountered:

jonesmz · 2020-02-19T00:42:43Z

I see in the netifd script that this is marked as a "TODO"

modemmanager_connected_method_static_ipv4() {
        local interface="$1"
        local wwan="$2"
        local address="$3"
        local prefix="$4"
        local gateway="$5"
        local mtu="$6"
        local dns1="$7"
        local dns2="$8"
        local metric="$9"

        local mask=""

        [ -n "${address}" ] || {
                proto_notify_error "${interface}" ADDRESS_MISSING
                return
        }

	[ -n "${prefix}" ] || {
                proto_notify_error "${interface}" PREFIX_MISSING
                return
        }

        mask=$(cdr2mask "${prefix}")

        # TODO: mtu reporting in proto handler

        proto_init_update "${wwan}" 1
        proto_set_keep 1
        echo "adding IPv4 address ${address}, netmask ${mask}"
        proto_add_ipv4_address "${address}" "${mask}"
        [ -n "${gateway}" ] && {
                echo "adding default IPv4 route via ${gateway}"
                proto_add_ipv4_route "0.0.0.0" "0" "${gateway}" "${address}"
        }
	[ -n "${dns1}" ] && {
                echo "adding primary DNS at ${dns1}"
                proto_add_dns_server "${dns1}"
        }
	[ -n "${dns2}" ] && {
                echo "adding secondary DNS at ${dns2}"
                proto_add_dns_server "${dns2}"
        }
	[ -n "$metric" ] && json_add_int metric "${metric}"
        proto_send_update "${interface}"
}

Any hints on how to implement this?

jonesmz · 2020-02-19T01:30:11Z

I think I've figured out what parts are missing.

As with most C language programs, Netifd is stunningly under documented, so I'm probably not fully understanding the complete picture.

That being said, here we go.

Netifd/interface-ip.c:

static const struct blobmsg_policy route_attr[__ROUTE_MAX] = {
	[ROUTE_INTERFACE] = { .name = "interface", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_TARGET] = { .name = "target", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_MASK] = { .name = "netmask", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_GATEWAY] = { .name = "gateway", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_METRIC] = { .name = "metric", .type = BLOBMSG_TYPE_INT32 },
	[ROUTE_MTU] = { .name = "mtu", .type = BLOBMSG_TYPE_INT32 },
	[ROUTE_TABLE] = { .name = "table", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_VALID] = { .name = "valid", .type = BLOBMSG_TYPE_INT32 },
	[ROUTE_SOURCE] = { .name = "source", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_ONLINK] = { .name = "onlink", .type = BLOBMSG_TYPE_BOOL },
	[ROUTE_TYPE] = { .name = "type", .type = BLOBMSG_TYPE_STRING },
	[ROUTE_PROTO] = { .name = "proto", .type = BLOBMSG_TYPE_STRING },
};

And then later in this function:

struct interface *
interface_ip_add_target_route(union if_addr *addr, bool v6, struct interface *iface)

We have this block:

	if ((cur = tb[ROUTE_MTU]) != NULL) {
	        route->mtu = blobmsg_get_u32(cur);
	        route->flags |= DEVROUTE_MTU;
	}

So there's already a ubus interface that has something to do with the MTU, and specifically it looks to be the ubus command message that tells netifd to set up the route.

Then, in netifd/netifd-functions.sh

proto_add_ipv4_route() {
        local target="$1"
        local mask="$2"
        local gw="$3"
        local source="$4"
        local metric="$5"

        append PROTO_ROUTE "$target/$mask/$gw/$metric///$source"
}

Which calls this function:

append() {
        local var="$1"
        local value="$2"
        local sep="${3:- }"

        eval "export -- \"$var=\${$var:+\${$var}\${value:+\$sep}}\$value\""
}

Which is probably the gnarlyiest function I've seen in a while. But my take is that it's appending values to the environment variable "PROTO_ROUTE" in a way that respects the '/' character as the seperator. Gnarly.

Then, some time later, somebody calls this function:

proto_send_update() {
        local interface="$1"

        proto_close_nested
        json_add_boolean keep "$PROTO_KEEP"
        _proto_push_array "ipaddr" "$PROTO_IPADDR" _proto_push_ipv4_addr
        _proto_push_array "ip6addr" "$PROTO_IP6ADDR" _proto_push_ipv6_addr
        _proto_push_array "routes" "$PROTO_ROUTE" _proto_push_route
        _proto_push_array "routes6" "$PROTO_ROUTE6" _proto_push_route
        _proto_push_array "ip6prefix" "$PROTO_PREFIX6" _proto_push_string
        _proto_push_array "dns" "$PROTO_DNS" _proto_push_string
        _proto_push_array "dns_search" "$PROTO_DNS_SEARCH" _proto_push_string
        _proto_notify "$interface"
}

Which calls this function:

_proto_push_route() {
        local str="$1";
        local target="${str%%/*}"
        str="${str#*/}"
        local mask="${str%%/*}"
        str="${str#*/}"
        local gw="${str%%/*}"
        str="${str#*/}"
        local metric="${str%%/*}"
        str="${str#*/}"
        local valid="${str%%/*}"
        str="${str#*/}"
        local table="${str%%/*}"
        str="${str#*/}"
        local source="${str}"

        json_add_object ""
        json_add_string target "$target"
        json_add_string netmask "$mask"
        [ -n "$gw" ] && json_add_string gateway "$gw"
        [ -n "$metric" ] && json_add_int metric "$metric"
        [ -n "$valid" ] && json_add_int valid "$valid"
        [ -n "$source" ] && json_add_string source "$source"
        [ -n "$table" ] && json_add_string table "$table"
        json_close_object
}

Which is the second gnarlyiest function I've seen in a while.

This function grabs all the details from the PROTO_ROUTE variable and sticks them into the current json object.

And then the resulting json string is sent to ubus like so:

_proto_notify() {
        local interface="$1"
        local options="$2"
        json_add_string "interface" "$interface"
        ubus $options call network.interface notify_proto "$(json_dump)"
}

So what appears to be missing is that despite netifd having direct support for this on the UBus interface, the associated scripts don't understand how to receive the MTU information from anything, or to communicate that to netifd.

I think that modifying

proto_add_ipv4_route

like such:

proto_add_ipv4_route() {
        local target="$1"
        local mask="$2"
        local gw="$3"
        local source="$4"
        local metric="$5"
        local mtu="$6"

        append PROTO_ROUTE "$target/$mask/$gw/$metric///$source/$mtu"
}

and then

_proto_push_route

_proto_push_route() {
        local str="$1";
        local target="${str%%/*}"
        str="${str#*/}"
        local mask="${str%%/*}"
        str="${str#*/}"
        local gw="${str%%/*}"
        str="${str#*/}"
        local metric="${str%%/*}"
        str="${str#*/}"
        local valid="${str%%/*}"
        str="${str#*/}"
        local table="${str%%/*}"
        str="${str#*/}"
        local source="${str%%/*}"
        str="${str#*/}"
        local mtu="${str}"

        json_add_object ""
        json_add_string target "$target"
        json_add_string netmask "$mask"
        [ -n "$gw" ] && json_add_string gateway "$gw"
        [ -n "$metric" ] && json_add_int metric "$metric"
        [ -n "$valid" ] && json_add_int valid "$valid"
        [ -n "$source" ] && json_add_string source "$source"
        [ -n "$table" ] && json_add_string table "$table"
        [ -n "$mtu" ] && json_add_string mtu "$mtu"
        json_close_object
}

Will fix the underlying netifd issue.

From there, you change :
modemmanager_connected_method_static_ipv4

Such that:

        [ -n "${gateway}" ] && {
                echo "adding default IPv4 route via ${gateway}"
                proto_add_ipv4_route "0.0.0.0" "0" "${gateway}" "${address}"
        }

Becomes

        [ -n "${gateway}" ] && {
                echo "adding default IPv4 route via ${gateway}"
                proto_add_ipv4_route "0.0.0.0" "0" "${gateway}" "${address}" "" "${mtu}"
        }

Unfortunately, at the moment, I don't know what to do about the missing "metric" variable. Theoretically passing just "" will be sufficient to make it get skipped, but I am not an expert on posix shell, so that's not clear yet.

That being said...

We do have the metric:

[ -n "$metric" ] && json_add_int metric "${metric}"

So why are we attempting to pass that as a loose integer variable in the json, instead of using the existing route function?

@aleksander0m Do you recall why you set up the metric like this?

jonesmz · 2020-02-19T01:56:12Z

@nickberry17 @aleksander0m ModemManager's ppp integration also does not set the MTU.

You'll see in /lib/netifd/proto/ppp.sh

proto_run_command "$config" /usr/sbin/pppd \
                nodetach ipparam "$config" \
                ifname "$pppname" \
                ${localip:+$localip:} \
                ${lcp_failure:+lcp-echo-interval $lcp_interval lcp-echo-failure $lcp_failure $lcp_adaptive} \
                ${ipv6:++ipv6} \
                ${autoipv6:+set AUTOIPV6=1} \
                ${ip6table:+set IP6TABLE=$ip6table} \
                ${peerdns:+set PEERDNS=$peerdns} \
                nodefaultroute \
                usepeerdns \
                $demand $persist maxfail $maxfail \
                ${holdoff:+holdoff "$holdoff"} \
                ${username:+user "$username" password "$password"} \
                ${connect:+connect "$connect"} \
                ${disconnect:+disconnect "$disconnect"} \
                ip-up-script /lib/netifd/ppp-up \
                ipv6-up-script /lib/netifd/ppp6-up \
                ip-down-script /lib/netifd/ppp-down \
                ipv6-down-script /lib/netifd/ppp-down \
                ${mtu:+mtu $mtu mru $mtu} \
                "$@" $pppd_options

Whereas modemmanager.sh

modemmanager_connected_method_ppp_ipv4() {
        local interface="$1"
        local ttyname="$2"
        local username="$3"
        local password="$4"

        proto_run_command "${interface}" /usr/sbin/pppd \
                "${ttyname}" \
                115200 \
                nodetach \
                noaccomp \
                nobsdcomp \
                nopcomp \
                novj \
                noauth \
                ${username:+ user $username} \
                ${password:+ password $password} \
                lcp-echo-failure 5 \
                lcp-echo-interval 15 \
                lock \
                crtscts \
                nodefaultroute \
                usepeerdns \
                ipparam "${interface}" \
                ip-up-script /lib/netifd/ppp-up \
                ip-down-script /lib/netifd/ppp-down
}

Does not attempt to communicate the mtu the bearer reports to pppd.

jonesmz · 2020-02-19T02:05:37Z

An attempt at a patch for the MTU problem (untested)

diff --git a/scripts/netifd-proto.sh b/scripts/netifd-proto.sh
index 87d337d..08b2d49 100644
--- a/scripts/netifd-proto.sh
+++ b/scripts/netifd-proto.sh
@@ -158,8 +158,9 @@ proto_add_ipv4_route() {
 	local gw="$3"
 	local source="$4"
 	local metric="$5"
+	local mtu="$6"

-	append PROTO_ROUTE "$target/$mask/$gw/$metric///$source"
+	append PROTO_ROUTE "$target/$mask/$gw/$metric///$source/$mtu"
 }

 proto_add_ipv6_route() {
@@ -170,8 +171,9 @@ proto_add_ipv6_route() {
 	local valid="$5"
 	local source="$6"
 	local table="$7"
+	local mtu="$8"

-	append PROTO_ROUTE6 "$target/$mask/$gw/$metric/$valid/$table/$source"
+	append PROTO_ROUTE6 "$target/$mask/$gw/$metric/$valid/$table/$source/$mtu"
 }

 proto_add_ipv6_prefix() {
@@ -288,7 +290,9 @@ _proto_push_route() {
 	str="${str#*/}"
 	local table="${str%%/*}"
 	str="${str#*/}"
-	local source="${str}"
+	local source="${str%%/*}"
+	str="${str#*/}"
+	local mtu="${str}"

 	json_add_object ""
 	json_add_string target "$target"
@@ -298,6 +302,7 @@ _proto_push_route() {
 	[ -n "$valid" ] && json_add_int valid "$valid"
 	[ -n "$source" ] && json_add_string source "$source"
 	[ -n "$table" ] && json_add_string table "$table"
+	[ -n "$mtu" ] && json_add_string mtu "$mtu"
 	json_close_object
 }

diff --git a/package/modemmanager/files/modemmanager.proto b/package/modemmanager/files/modemmanager.proto
index 536073d..408e611 100644
--- a/package/modemmanager/files/modemmanager.proto
+++ b/package/modemmanager/files/modemmanager.proto
@@ -116,6 +116,7 @@ modemmanager_connected_method_ppp_ipv4() {
 	local ttyname="$2"
 	local username="$3"
 	local password="$4"
+	local mtu="$5"

 	proto_run_command "${interface}" /usr/sbin/pppd \
 	        "${ttyname}" \
@@ -136,7 +137,8 @@ modemmanager_connected_method_ppp_ipv4() {
 	        usepeerdns \
 	        ipparam "${interface}" \
 	        ip-up-script /lib/netifd/ppp-up \
-	        ip-down-script /lib/netifd/ppp-down
+	        ip-down-script /lib/netifd/ppp-down \
+	        ${mtu:+mtu $mtu mru $mtu}
 }

 modemmanager_disconnected_method_ppp_ipv4() {
@@ -207,15 +209,13 @@ modemmanager_connected_method_static_ipv4() {

 	mask=$(cdr2mask "${prefix}")

-	# TODO: mtu reporting in proto handler
-
 	proto_init_update "${wwan}" 1
 	proto_set_keep 1
 	echo "adding IPv4 address ${address}, netmask ${mask}"
 	proto_add_ipv4_address "${address}" "${mask}"
 	[ -n "${gateway}" ] && {
 	        echo "adding default IPv4 route via ${gateway}"
-	        proto_add_ipv4_route "0.0.0.0" "0" "${gateway}" "${address}"
+	        proto_add_ipv4_route "0.0.0.0" "0" "${gateway}" "${address}" "${metric}" "${mtu}"
 	}
 	[ -n "${dns1}" ] && {
 	        echo "adding primary DNS at ${dns1}"
@@ -225,7 +225,6 @@ modemmanager_connected_method_static_ipv4() {
 	        echo "adding secondary DNS at ${dns2}"
 	        proto_add_dns_server "${dns2}"
 	}
-	[ -n "$metric" ] && json_add_int metric "${metric}"
 	proto_send_update "${interface}"
 }

@@ -270,8 +269,6 @@ modemmanager_connected_method_static_ipv6() {
 	        return
 	}

-	# TODO: mtu reporting in proto handler
-
 	proto_init_update "${wwan}" 1
 	proto_set_keep 1
 	echo "adding IPv6 address ${address}, prefix ${prefix}"
@@ -280,7 +277,7 @@ modemmanager_connected_method_static_ipv6() {
 	[ -n "${gateway}" ] && {
 	        echo "adding default IPv6 route via ${gateway}"
 	        proto_add_ipv6_route "${gateway}" "128"
-	        proto_add_ipv6_route "::0" "0" "${gateway}" "" "" "${address}/${prefix}"
+	        proto_add_ipv6_route "::0" "0" "${gateway}" "" "" "${address}/${prefix}" "${metric}" "${mtu}"
 	}
 	[ -n "${dns1}" ] && {
 	        echo "adding primary DNS at ${dns1}"
@@ -290,7 +287,6 @@ modemmanager_connected_method_static_ipv6() {
 	        echo "adding secondary DNS at ${dns2}"
 	        proto_add_dns_server "${dns2}"
 	}
-	[ -n "$metric" ] && json_add_int metric "${metric}"
 	proto_send_update "${interface}"
 }

@@ -412,7 +408,7 @@ proto_modemmanager_setup() {
 		                modemmanager_connected_method_static_ipv4 "${interface}" "${beareriface}" "${address}" "${prefix}" "${gateway}" "${mtu}" "${dns1}" "${dns2}" "${metric}"
 		                ;;
 	        "ppp")
-		                modemmanager_connected_method_ppp_ipv4 "${interface}" "${beareriface}" "${username}" "${password}"
+		                modemmanager_connected_method_ppp_ipv4 "${interface}" "${beareriface}" "${username}" "${password}" "${mtu}"
 		                ;;
 	        *)
 		                proto_notify_error "${interface}" UNKNOWN_METHOD

jonesmz · 2020-02-19T04:08:57Z

Well, with those two patches, i get this ubus call:

ubus call network.interface notify_proto "{ "action": 0, "ifname": "wwan0", "link-up": true, "keep": true, "ipaddr": [ { "ipaddr": "10.3.129.231", "mask": "255.255.255.240" } ], "routes": [ { "target": "0.0.0.0", "netmask": "0", "gateway": "10.3.129.232", "source": "10.3.129.231", "mtu": "1430" } ], "dns": [ "172.26.38.1", "10.0.0.252" ], "interface": "wwan" }"

Unfortunately, it doesn't result in the MTU changing.

Probably because my version of netifd is from June 2018, and I see several MTU related changes in the git history since then.

Will try updating to the 19.07 version and see what happens.

jonesmz · 2020-02-19T07:15:01Z

Nope. netifd from openwrt-19.07 makes no difference.

jonesmz · 2020-02-19T07:15:35Z

I'm not going to bother trying to patch the netifd c code.

i'll just add a script that checks the mtu from modemmanager and sets it with the ifconfig program.

jonesmz · 2020-05-14T18:36:57Z

I just want to be clear, to make sure everyone is on the same page:

I am not working on this any more.
I am not planning to make a pull request, or identify specific code changes that need to be made.

I've completely worked-around this problem with a fugly hack that sets the MTU after modemmanager and netifd finish their work, by looping over modemmanager's mmcli output and checking against the mtu of the interface, and setting it with ifconfig if needed.

Since that solves my problem good enough, I'll stick with that.

That being said, if anyone else wants to jump in on this, I would be happy to collaborate, including testing, code review, and programming help.

aleksander0m · 2020-05-15T08:17:32Z

We do have the metric:
[ -n "$metric" ] && json_add_int metric "${metric}"
So why are we attempting to pass that as a loose integer variable in the json, instead of using the existing route function?

@aleksander0m Do you recall why you set up the metric like this?

I don't remember, but if I had to give an answer, I guess it's because the MTU is a per-interface configuration, not a route configuration; isn't that right? Why is the MTU part of the route settings?

jonesmz · 2020-05-15T17:02:32Z

Apologies for any confusion.

My question about the metric was a code style question that arose while investigating the MTU. Not really related to the MTU.

MTU is a per interface thing, not a per route thing, as you said.

jonesmz · 2020-05-15T17:50:51Z

Re-reading this whole issue, I see why you ask about the MTU and the route.

Yes, you're correct, and I was also confused why it looked like netifd wanted to be told about the MTU as part of the route.

The Netifd source code is not documented sufficiently for me to understand how to fix this in the amount of time I have available for the task, so I gave up and implemented a work-around.

But I agree with you that if the MTU is to be set, it should be set on the interface itself, not on the route.

Please also note that the MTU is not being passed to pppd either, so that's something to consider and probably would be easy enough to address quickly, which would leave only the static ip address case as not having the MTU (I think? DHCP should get it automatically, if I'm not mistaken)

amollad · 2020-05-18T11:39:39Z

I've completely worked-around this problem with a fugly hack that sets the MTU after modemmanager and netifd finish their work, by looping over modemmanager's mmcli output and checking against the mtu of the interface, and setting it with ifconfig if needed.

@jonesmz I'm also new to netifd. How do you know when "netifd" finished it's work? i.e. from where you call your workaround?

jonesmz · 2020-05-18T17:28:24Z

You would do something like this

bearerpath="$(modemmanager_get_field "${modemstatus}" "modem.generic.bearers.value\[1\]")"
bearerstatus="$(mmcli --bearer="${bearerpath}" --output-keyvalue)"
bearerinterface="$(modemmanager_get_field "${bearerstatus}" "bearer.status.interface")"
bearermtu="$(modemmanager_get_field "${bearerstatus}" "bearer.ipv4-config.mtu")"
if	[ ! -z "${bearermtu}" ] &&
	[ ! -z "${bearerinterface}" ] &&
	[ "${bearermtu}" != "--" ] &&
	[ "${bearermtu}" != "$(cat /sys/class/net/${bearerinterface}/mtu)" ]
then
	echo "Adjusting the MTU of ${bearerinterface} to ${bearermtu}"
	ifconfig ${bearerinterface} mtu ${bearermtu}
fi

I just loop on that forever, with a sleep per iteration.

It's hacky, but it works.

amollad · 2020-05-19T10:45:55Z

@jonesmz Thank you for your investigation. I looked into your proposed changes and the MTU which netifd modifies is "route MTU" and not "interface MTU". So after applying your suggested changes we see:

$ ip route show
default via 25.225.105.222 dev wwan0 proto static src 25.225.105.221 mtu 1430

Please note "mtu 1430" in the end.

Same for ipv6

jonesmz · 2020-05-19T16:31:05Z

Well honestly I didn't even know a route MTU was a thing.

Since you are new to netifd, I suppose you probably don't know of any mechanism to set the MTU for the interface?

amollad · 2020-05-20T00:49:09Z

Well honestly I didn't even know a route MTU was a thing.

Since you are new to netifd, I suppose you probably don't know of any mechanism to set the MTU for the interface?

Well, only method I found to set interface MTU via netifd is to have below UCI in /etc/config/network. I'm not able to find any ubus call

config device
option name wwan0
option mtu 1430
option mtu6 1430

Using the same method used by other protocol handlers like uqmi. Fixes openwrt#11383 Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

Using the same method used by other protocol handlers like uqmi. Fixes openwrt#11383 Signed-off-by: Aleksander Morgado <aleksander@aleksander.es> (cherry picked from commit 41552c1)

Using the same method used by other protocol handlers like uqmi. Fixes openwrt/packages#11383 Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

Using the same method used by other protocol handlers like uqmi. Fixes openwrt#11383 Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

aleksander0m mentioned this issue May 25, 2020

modemmanager: set interface MTU based on bearer settings #12302

Merged

neheb closed this as completed in #12302 May 25, 2020

aleksander0m mentioned this issue May 26, 2020

modemmanager: set interface MTU based on bearer settings #12313

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ModemManager: MTU not being set based on Bearer info #11383

ModemManager: MTU not being set based on Bearer info #11383

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented May 14, 2020

aleksander0m commented May 15, 2020

jonesmz commented May 15, 2020 •

edited

Loading

jonesmz commented May 15, 2020

amollad commented May 18, 2020 •

edited

Loading

jonesmz commented May 18, 2020

amollad commented May 19, 2020 •

edited

Loading

jonesmz commented May 19, 2020

amollad commented May 20, 2020

ModemManager: MTU not being set based on Bearer info #11383

ModemManager: MTU not being set based on Bearer info #11383

Comments

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented Feb 19, 2020

jonesmz commented May 14, 2020

aleksander0m commented May 15, 2020

jonesmz commented May 15, 2020 • edited Loading

jonesmz commented May 15, 2020

amollad commented May 18, 2020 • edited Loading

jonesmz commented May 18, 2020

amollad commented May 19, 2020 • edited Loading

jonesmz commented May 19, 2020

amollad commented May 20, 2020

jonesmz commented May 15, 2020 •

edited

Loading

amollad commented May 18, 2020 •

edited

Loading

amollad commented May 19, 2020 •

edited

Loading