Skip to content
This repository has been archived by the owner on Jul 25, 2022. It is now read-only.

Integrate network interface data into IML #2353

Merged
merged 4 commits into from
Nov 6, 2020
Merged

Integrate network interface data into IML #2353

merged 4 commits into from
Nov 6, 2020

Conversation

johnsonw
Copy link
Contributor

@johnsonw johnsonw commented Oct 28, 2020 •
edited by jgrund
Loading

Now that the network logic / parsing lives on the rust side in the
iml-agent, the next step is to:

  1. Create a daemon plugin to fetch the data
  2. Ingest the data on the manager side:
    a. Insert the interface data into the database
    b. Insert the interface stats (rx/tx bytes) into influx

Signed-off-by: johnsonw wjohnson@whamcloud.com

This change is Reviewable

@johnsonw johnsonw self-assigned this Oct 28, 2020
@johnsonw johnsonw force-pushed the linux-devices branch 3 times, most recently from 2f25c46 to fa37600 Compare November 2, 2020 16:25
@johnsonw johnsonw force-pushed the integrate-network-interfaces branch 2 times, most recently from ab32d54 to 92352c8 Compare November 2, 2020 16:39
@johnsonw johnsonw marked this pull request as ready for review November 2, 2020 16:43
@johnsonw
Copy link
Contributor Author

johnsonw commented Nov 2, 2020

Output from an RPM based install:

chroma=> select * from network_interface;
    mac_address    | name |  inet4_address   |         inet6_address          | lnd_type | state_up | host_id
-------------------+------+------------------+--------------------------------+----------+----------+---------
 52:54:00:fd:32:21 | eth0 | {10.0.2.15/24}   | {fe80::5054:ff:fefd:3221/64}   | tcp      | t        |       4
 08:00:27:fb:c3:f3 | eth1 | {10.73.20.22/24} | {fe80::a00:27ff:fefb:c3f3/64}  | tcp      | t        |       4
 08:00:27:c2:01:de | eth2 | {10.73.10.22/24} | {fe80::a00:27ff:fec2:1de/64}   | tcp      | t        |       4
 08:00:27:9f:3a:7f | eth3 | {10.73.40.22/24} | {fe80::a00:27ff:fe9f:3a7f/64}  | tcp      | t        |       4
 08:00:27:7f:e1:88 | eth4 | {10.73.50.22/24} | {fe80::a00:27ff:fe7f:e188/64}  | tcp      | t        |       4
 08:00:27:2d:73:73 | eth5 | {}               | {}                             | tcp      | t        |       4
 08:00:27:c3:5f:ab | eth1 | {10.73.20.21/24} | {fe80::a00:27ff:fec3:5fab/64}  | tcp      | t        |       3
 08:00:27:3a:19:5f | eth2 | {10.73.10.21/24} | {fe80::a00:27ff:fe3a:195f/64}  | tcp      | t        |       3
 08:00:27:d2:f3:88 | eth3 | {10.73.40.21/24} | {fe80::a00:27ff:fed2:f388/64}  | tcp      | t        |       3
 08:00:27:7d:32:b5 | eth4 | {10.73.50.21/24} | {fe80::a00:27ff:fe7d:32b5/64}  | tcp      | t        |       3
 08:00:27:12:9a:e6 | eth5 | {}               | {}                             | tcp      | t        |       3
 08:00:27:34:14:6a | eth1 | {10.73.20.12/24} | {fe80::a00:27ff:fe34:146a/64}  | tcp      | t        |       2
 08:00:27:42:09:29 | eth2 | {10.73.10.12/24} | {fe80::a00:27ff:fe42:929/64}   | tcp      | t        |       2
 08:00:27:67:8c:19 | eth3 | {10.73.40.12/24} | {fe80::a00:27ff:fe67:8c19/64}  | tcp      | t        |       2
 08:00:27:e4:37:78 | eth4 | {10.73.50.12/24} | {fe80::a00:27ff:fee4:3778/64}  | tcp      | t        |       2
 08:00:27:ba:5a:b1 | eth5 | {}               | {fe80::a9f2:8495:9f7c:c6fc/64} | tcp      | t        |       2
 08:00:27:18:c3:14 | eth1 | {10.73.20.11/24} | {fe80::a00:27ff:fe18:c314/64}  | tcp      | t        |       1
 08:00:27:1c:67:e3 | eth2 | {10.73.10.11/24} | {fe80::a00:27ff:fe1c:67e3/64}  | tcp      | t        |       1
 08:00:27:b8:8c:c3 | eth3 | {10.73.40.11/24} | {fe80::a00:27ff:feb8:8cc3/64}  | tcp      | t        |       1
 08:00:27:bc:a0:d9 | eth4 | {10.73.50.11/24} | {fe80::a00:27ff:febc:a0d9/64}  | tcp      | t        |       1
 08:00:27:55:46:56 | eth5 | {}               | {}                             | tcp      | t        |       1


select * from net group by host_id,interface limit 1;
name: net
tags: host_id=1, interface=eth0
time                rx_bytes  tx_bytes
----                --------  --------
1604327385895183131 197634544 1888705
name: net
tags: host_id=1, interface=eth1
time                rx_bytes  tx_bytes
----                --------  --------
1604327385895183131 347257094 319464180
name: net
tags: host_id=1, interface=eth2
time                rx_bytes   tx_bytes
----                --------   --------
1604327385895183131 1052575714 1712010943
name: net
tags: host_id=1, interface=eth3
time                rx_bytes  tx_bytes
----                --------  --------
1604327385895183131 277425100 735596138
name: net
tags: host_id=1, interface=eth4
time                rx_bytes  tx_bytes
----                --------  --------
1604327385895183131 251295938 21405358
name: net
tags: host_id=1, interface=eth5
time                rx_bytes tx_bytes
----                -------- --------
1604327385895183131 3612204  5405598
name: net
tags: host_id=2, interface=eth0
time                rx_bytes  tx_bytes
----                --------  --------
1604327436139280796 197546036 1857174
name: net
tags: host_id=2, interface=eth1
time                rx_bytes  tx_bytes
----                --------  --------
1604327436139280796 593745640 524759622
name: net
tags: host_id=2, interface=eth2
time                rx_bytes   tx_bytes
----                --------   --------
1604327436139280796 1052074712 1712081866
name: net
tags: host_id=2, interface=eth3
time                rx_bytes  tx_bytes
----                --------  --------
1604327436139280796 278280446 944336402
name: net
tags: host_id=2, interface=eth4
time                rx_bytes  tx_bytes
----                --------  --------
1604327436139280796 250913716 21234574
name: net
tags: host_id=2, interface=eth5
time                rx_bytes tx_bytes
----                -------- --------
1604327436139280796 3622122  5387834
name: net
tags: host_id=3, interface=eth0
time                rx_bytes  tx_bytes
----                --------  --------
1604327475523798720 197471664 1692929
name: net
tags: host_id=3, interface=eth1
time                rx_bytes  tx_bytes
----                --------  --------
1604327475523798720 347945850 415185586
name: net
tags: host_id=3, interface=eth2
time                rx_bytes  tx_bytes
----                --------  --------
1604327475523798720 953636684 1595886982
name: net
tags: host_id=3, interface=eth3
time                rx_bytes   tx_bytes
----                --------   --------
1604327475523798720 1238276882 779585792
name: net
tags: host_id=3, interface=eth4
time                rx_bytes   tx_bytes
----                --------   --------
1604327475523798720 1347680520 1738452332
name: net
tags: host_id=3, interface=eth5
time                rx_bytes tx_bytes
----                -------- --------
1604327475523798720 3610836  5417930
name: net
tags: host_id=4, interface=eth0
time                rx_bytes  tx_bytes
----                --------  --------
1604327514494937076 197595635 1654979
name: net
tags: host_id=4, interface=eth1
time                rx_bytes  tx_bytes
----                --------  --------
1604327514494937076 354408270 420803928
name: net
tags: host_id=4, interface=eth2
time                rx_bytes  tx_bytes
----                --------  --------
1604327514494937076 970137506 1623340852
name: net
tags: host_id=4, interface=eth3
time                rx_bytes   tx_bytes
----                --------   --------
1604327514494937076 1436315916 2240694958
name: net
tags: host_id=4, interface=eth4
time                rx_bytes   tx_bytes
----                --------   --------
1604327514494937076 1162755148 294222404
name: net
tags: host_id=4, interface=eth5
time                rx_bytes tx_bytes
----                -------- --------
1604327514494937076 3635118  5388176

Output from a docker install:

chroma=# select * from network_interface
chroma-# ;
    mac_address    | name |  inet4_address   |         inet6_address          | lnd_type | state_up | host_id
-------------------+------+------------------+--------------------------------+----------+----------+---------
 08:00:27:c3:5f:ab | eth1 | {10.73.20.21/24} | {fe80::a00:27ff:fec3:5fab/64}  | tcp      | t        |       3
 08:00:27:3a:19:5f | eth2 | {10.73.10.21/24} | {fe80::a00:27ff:fe3a:195f/64}  | tcp      | t        |       3
 08:00:27:d2:f3:88 | eth3 | {10.73.40.21/24} | {fe80::a00:27ff:fed2:f388/64}  | tcp      | t        |       3
 08:00:27:7d:32:b5 | eth4 | {10.73.50.21/24} | {fe80::a00:27ff:fe7d:32b5/64}  | tcp      | t        |       3
 08:00:27:12:9a:e6 | eth5 | {}               | {}                             | tcp      | t        |       3
 08:00:27:fb:c3:f3 | eth1 | {10.73.20.22/24} | {fe80::a00:27ff:fefb:c3f3/64}  | tcp      | t        |       4
 08:00:27:c2:01:de | eth2 | {10.73.10.22/24} | {fe80::a00:27ff:fec2:1de/64}   | tcp      | t        |       4
 08:00:27:9f:3a:7f | eth3 | {10.73.40.22/24} | {fe80::a00:27ff:fe9f:3a7f/64}  | tcp      | t        |       4
 08:00:27:7f:e1:88 | eth4 | {10.73.50.22/24} | {fe80::a00:27ff:fe7f:e188/64}  | tcp      | t        |       4
 08:00:27:2d:73:73 | eth5 | {}               | {}                             | tcp      | t        |       4
 08:00:27:18:c3:14 | eth1 | {10.73.20.11/24} | {fe80::a00:27ff:fe18:c314/64}  | tcp      | t        |       1
 08:00:27:1c:67:e3 | eth2 | {10.73.10.11/24} | {fe80::a00:27ff:fe1c:67e3/64}  | tcp      | t        |       1
 08:00:27:b8:8c:c3 | eth3 | {10.73.40.11/24} | {fe80::a00:27ff:feb8:8cc3/64}  | tcp      | t        |       1
 08:00:27:bc:a0:d9 | eth4 | {10.73.50.11/24} | {fe80::a00:27ff:febc:a0d9/64}  | tcp      | t        |       1
 08:00:27:55:46:56 | eth5 | {}               | {}                             | tcp      | t        |       1
 52:54:00:fd:32:21 | eth0 | {10.0.2.15/24}   | {fe80::5054:ff:fefd:3221/64}   | tcp      | t        |       2
 08:00:27:34:14:6a | eth1 | {10.73.20.12/24} | {fe80::a00:27ff:fe34:146a/64}  | tcp      | t        |       2
 08:00:27:42:09:29 | eth2 | {10.73.10.12/24} | {fe80::a00:27ff:fe42:929/64}   | tcp      | t        |       2
 08:00:27:67:8c:19 | eth3 | {10.73.40.12/24} | {fe80::a00:27ff:fe67:8c19/64}  | tcp      | t        |       2
 08:00:27:e4:37:78 | eth4 | {10.73.50.12/24} | {fe80::a00:27ff:fee4:3778/64}  | tcp      | t        |       2
 08:00:27:ba:5a:b1 | eth5 | {}               | {fe80::a9f2:8495:9f7c:c6fc/64} | tcp      | t        |       2
(21 rows)

image

Base automatically changed from linux-devices to master November 3, 2020 14:14
@johnsonw
Integrate network interface data into IML
24e029b 
Now that the network logic / parsing lives on the rust side in the
iml-agent, the next step is to:
1. Create a daemon plugin to fetch the data
2. Ingest the data on the manager side:
   a. Insert the interface data into the database
   b. Insert the interface stats (rx/tx bytes) into influx

Signed-off-by: johnsonw <wjohnson@whamcloud.com>
@johnsonw
Add missing copyright
5049206 
Signed-off-by: johnsonw <wjohnson@whamcloud.com>
@johnsonw johnsonw requested a review from a team November 3, 2020 22:19
jgrund
jgrund reviewed Nov 4, 2020
View reviewed changes
rust-iml.spec Outdated Show resolved Hide resolved
@jgrund jgrund self-requested a review November 4, 2020 14:57
@johnsonw
CR updates
9436eab 
Signed-off-by: johnsonw <wjohnson@whamcloud.com>
@johnsonw
Revert tokio update
c75c357 
Signed-off-by: johnsonw <wjohnson@whamcloud.com>
@jgrund jgrund requested a review from ip1981 November 5, 2020 21:42
jgrund
jgrund reviewed Nov 5, 2020
View reviewed changes
iml-agent/src/daemon_plugins/network.rs
Comment on lines +27 to +30
let xs = get_interfaces().await?;
let xs = serde_json::to_value(xs).map(Some)?;

Ok(xs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this could all just be inside update_session.

@jgrund jgrund merged commit c367c64 into master Nov 6, 2020
@jgrund jgrund deleted the integrate-network-interfaces branch November 6, 2020 13:12
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jgrund jgrund jgrund approved these changes

@nlinker nlinker nlinker approved these changes

@ip1981 ip1981 Awaiting requested review from ip1981

Assignees

@johnsonw johnsonw

Labels
enhancement
Projects
None yet
Milestone
No milestone
4 participants
@johnsonw @ip1981 @jgrund @nlinker