Too many timeouts when working with high concurrency #656

scanpatch · 2019-09-18T23:04:18Z

I use fasthttp client in an application that collects information about millions of sites on the network.
To do this really quickly and in parallel, I create a bunch of goroutines in which I execute c.httpClient.DoTimeout (...) requests

If I run no more than ~ 100 threads per core, then I successfully receive answers for all requests.
If I run more than 100 threads per core, then part of the requests will be interrupted by timeout and I will get errors

The problem is definitely not in the sites themselves.

I think there are some restrictions on the number of open connections or something like that, but I don’t know where to dig.

I will be very glad to any prompts.

erikdubbelboer · 2019-09-20T15:13:05Z

What kind of CPU are you running on?
What is the CPU usage once you see the many timeouts?
I assume this is Linux, correct?
What did you set ulimit -n to?

scanpatch · 2019-09-20T21:26:44Z

@erikdubbelboer ,

CPU details:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               142
Model name:          Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
Stepping:            11
CPU MHz:             800.039
CPU max MHz:         3900,0000
CPU min MHz:         400,0000
BogoMIPS:            3600.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-7

_* The application was launched on different servers with a variety of CPUs. The result is always the same.

CPU usage

top - 00:07:30 up  2:08,  3 users,  load average: 0,55, 1,04, 1,17
Tasks: 361 total,   1 running, 289 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8,8 us,  3,1 sy,  0,0 ni, 86,6 id,  0,1 wa,  0,0 hi,  1,4 si,  0,0 st
KiB Mem : 16261832 total,  8130404 free,  3775072 used,  4356356 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used. 11479040 avail Mem

htop screenshot:

Graphs of processor and memory usage from the time the program starts until the timeouts appear:

OS: Ubuntu 18.04.3 LTS

ulimit -n: 65000

kirillDanshin · 2019-09-21T07:56:56Z

@r-romanov
is this the actual CPU specs from your production? AFAIK this is a mobile CPU like for in some HP laptops for example

can we see uname -a and sysctl -a please?
also, can you please tell us your timeout settings? a minimal snippet that triggers the issue would be awesome.

iapyeh · 2019-09-21T08:23:25Z

It seems that the application is of large amount of concurrent "clients". Would It be short of ports, since max port is 65535?

kirillDanshin · 2019-09-21T08:45:04Z

@iapyeh actually I don't think so, if we speaking of let's say 150 requests * 8 cores it would be much less than 65535

erikdubbelboer · 2019-09-21T12:14:07Z

@r-romanov what kind of errors are you getting exactly? I mean the returned error of client.DoTimeout().

Do you have a list of the urls you are trying to fetch somewhere? I'm trying to reproduce this but I'm not sure which urls to fetch. Are all urls on different domains or are they on the same domain?

If they are on different domains try adding request.SetConnectionClose() so you don't have keep-alive connections open to domains that only receive a single request.

scanpatch · 2019-09-22T09:50:34Z

@kirillDanshin,
This is the CPU of my personal PC, on which I am developing. Not production system.

uname -a:
Linux lesya-HP-ProBook-440-G6 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

sysctl -a

abi.vsyscall32 = 1
debug.exception-trace = 1
debug.kprobes-optimization = 1
dev.cdrom.autoclose = 1
dev.cdrom.autoeject = 0
dev.cdrom.check_media = 0
dev.cdrom.debug = 0
dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17
dev.cdrom.info = 
dev.cdrom.info = drive name:	
dev.cdrom.info = drive speed:	
dev.cdrom.info = drive # of slots:
dev.cdrom.info = Can close tray:	
dev.cdrom.info = Can open tray:	
dev.cdrom.info = Can lock tray:	
dev.cdrom.info = Can change speed:
dev.cdrom.info = Can select disk:
dev.cdrom.info = Can read multisession:
dev.cdrom.info = Can read MCN:	
dev.cdrom.info = Reports media changed:
dev.cdrom.info = Can play audio:	
dev.cdrom.info = Can write CD-R:	
dev.cdrom.info = Can write CD-RW:
dev.cdrom.info = Can read DVD:	
dev.cdrom.info = Can write DVD-R:
dev.cdrom.info = Can write DVD-RAM:
dev.cdrom.info = Can read MRW:	
dev.cdrom.info = Can write MRW:	
dev.cdrom.info = Can write RAM:	
dev.cdrom.info = 
dev.cdrom.info = 
dev.cdrom.lock = 0
dev.hpet.max-user-freq = 64
dev.i915.oa_max_sample_rate = 100000
dev.i915.perf_stream_paranoid = 1
dev.mac_hid.mouse_button2_keycode = 97
dev.mac_hid.mouse_button3_keycode = 100
dev.mac_hid.mouse_button_emulation = 0
dev.parport.default.spintime = 500
dev.parport.default.timeslice = 200
dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000
dev.scsi.logging_level = 0
dev.tty.ldisc_autoload = 1
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.binfmt_misc.cli = enabled
fs.binfmt_misc.cli = interpreter /usr/lib/binfmt-support/run-detectors
fs.binfmt_misc.cli = flags: 
fs.binfmt_misc.cli = offset 0
fs.binfmt_misc.cli = magic 4d5a
fs.binfmt_misc.python2/7 = enabled
fs.binfmt_misc.python2/7 = interpreter /usr/bin/python2.7
fs.binfmt_misc.python2/7 = flags: 
fs.binfmt_misc.python2/7 = offset 0
fs.binfmt_misc.python2/7 = magic 03f30d0a
fs.binfmt_misc.python3/7 = enabled
fs.binfmt_misc.python3/7 = interpreter /usr/bin/python3.7
fs.binfmt_misc.python3/7 = flags: 
fs.binfmt_misc.python3/7 = offset 0
fs.binfmt_misc.python3/7 = magic 420d0d0a
fs.binfmt_misc.python3/6 = enabled
fs.binfmt_misc.python3/6 = interpreter /usr/bin/python3.6
fs.binfmt_misc.python3/6 = flags: 
fs.binfmt_misc.python3/6 = offset 0
fs.binfmt_misc.python3/6 = magic 330d0d0a
fs.binfmt_misc.status = enabled
fs.dentry-state = 289192	248547	45	0	35161	0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 3320627
fs.file-max = 1616756
fs.file-nr = 18995	0	1616756
fs.inode-nr = 253800	614
fs.inode-state = 253800	614	0	0	0	0	0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288
fs.lease-break-time = 45
fs.leases-enable = 1
fs.mount-max = 100000
fs.mqueue.msg_default = 10
fs.mqueue.msg_max = 10
fs.mqueue.msgsize_default = 8192
fs.mqueue.msgsize_max = 8192
fs.mqueue.queues_max = 256
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.pipe-user-pages-hard = 0
fs.pipe-user-pages-soft = 16384
fs.protected_fifos = 0
fs.protected_hardlinks = 1
fs.protected_regular = 0
fs.protected_symlinks = 1
fs.quota.allocated_dquots = 0
fs.quota.cache_hits = 0
fs.quota.drops = 0
fs.quota.free_dquots = 0
fs.quota.lookups = 0
fs.quota.reads = 0
fs.quota.syncs = 116
fs.quota.writes = 0
fs.suid_dumpable = 2
fs.xfs.error_level = 3
fs.xfs.filestream_centisecs = 3000
fs.xfs.inherit_noatime = 1
fs.xfs.inherit_nodefrag = 1
fs.xfs.inherit_nodump = 1
fs.xfs.inherit_nosymlinks = 0
fs.xfs.inherit_sync = 1
fs.xfs.irix_sgid_inherit = 0
fs.xfs.irix_symlink_mode = 0
fs.xfs.panic_mask = 0
fs.xfs.rotorstep = 1
fs.xfs.speculative_cow_prealloc_lifetime = 1800
fs.xfs.speculative_prealloc_lifetime = 300
fs.xfs.stats_clear = 0
fs.xfs.xfssyncd_centisecs = 3000
kernel.acct = 4	2	30
kernel.acpi_video_flags = 0
kernel.auto_msgmni = 0
kernel.bootloader_type = 114
kernel.bootloader_version = 2
kernel.cad_pid = 1
kernel.cap_last_cap = 37
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %d %P
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 0
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = (none)
kernel.firmware_config.force_sysfs_fallback = 0
kernel.firmware_config.ignore_sysfs_fallback = 0
kernel.ftrace_dump_on_oops = 0
kernel.ftrace_enabled = 1
kernel.hardlockup_all_cpu_backtrace = 0
kernel.hardlockup_panic = 0
kernel.hostname = lesya-HP-ProBook-440-G6
kernel.hotplug = 
kernel.hung_task_check_count = 4194304
kernel.hung_task_check_interval_secs = 0
kernel.hung_task_panic = 0
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = 10
kernel.io_delay_type = 1
kernel.kexec_load_disabled = 0
kernel.keys.gc_delay = 300
kernel.keys.maxbytes = 20000
kernel.keys.maxkeys = 200
kernel.keys.persistent_keyring_expiry = 259200
kernel.keys.root_maxbytes = 25000000
kernel.keys.root_maxkeys = 1000000
kernel.kptr_restrict = 1
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msg_next_id = -1
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 32000
kernel.ngroups_max = 65536
kernel.nmi_watchdog = 1
kernel.ns_last_pid = 25091
kernel.numa_balancing = 0
kernel.numa_balancing_scan_delay_ms = 1000
kernel.numa_balancing_scan_period_max_ms = 60000
kernel.numa_balancing_scan_period_min_ms = 1000
kernel.numa_balancing_scan_size_mb = 256
kernel.osrelease = 5.0.0-29-generic
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 0
kernel.panic_on_io_nmi = 0
kernel.panic_on_oops = 0
kernel.panic_on_rcu_stall = 0
kernel.panic_on_unrecovered_nmi = 0
kernel.panic_on_warn = 0
kernel.panic_print = 0
kernel.perf_cpu_time_max_percent = 25
kernel.perf_event_max_contexts_per_stack = 8
kernel.perf_event_max_sample_rate = 31750
kernel.perf_event_max_stack = 127
kernel.perf_event_mlock_kb = 516
kernel.perf_event_paranoid = 3
kernel.pid_max = 32768
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 4	4	1	7
kernel.printk_delay = 0
kernel.printk_devkmsg = ratelimit
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 7
kernel.pty.reserve = 1024
kernel.random.boot_id = cbf897c8-b0dc-40f7-862c-fd434d720419
kernel.random.entropy_avail = 3773
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.urandom_min_reseed_secs = 60
kernel.random.uuid = 7af06e7f-0717-4140-918e-be7c3cdd2a16
kernel.random.write_wakeup_threshold = 896
kernel.randomize_va_space = 2
kernel.real-root-dev = 0
kernel.sched_autogroup_enabled = 1
kernel.sched_cfs_bandwidth_slice_us = 5000
kernel.sched_child_runs_first = 0
kernel.sched_domain.cpu0.domain0.busy_factor = 32
kernel.sched_domain.cpu0.domain0.busy_idx = 0
kernel.sched_domain.cpu0.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu0.domain0.flags = 4783
kernel.sched_domain.cpu0.domain0.forkexec_idx = 0
kernel.sched_domain.cpu0.domain0.idle_idx = 0
kernel.sched_domain.cpu0.domain0.imbalance_pct = 110
kernel.sched_domain.cpu0.domain0.max_interval = 4
kernel.sched_domain.cpu0.domain0.max_newidle_lb_cost = 29737
kernel.sched_domain.cpu0.domain0.min_interval = 2
kernel.sched_domain.cpu0.domain0.name = SMT
kernel.sched_domain.cpu0.domain0.newidle_idx = 0
kernel.sched_domain.cpu0.domain0.wake_idx = 0
kernel.sched_domain.cpu0.domain1.busy_factor = 32
kernel.sched_domain.cpu0.domain1.busy_idx = 2
kernel.sched_domain.cpu0.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu0.domain1.flags = 4655
kernel.sched_domain.cpu0.domain1.forkexec_idx = 0
kernel.sched_domain.cpu0.domain1.idle_idx = 0
kernel.sched_domain.cpu0.domain1.imbalance_pct = 117
kernel.sched_domain.cpu0.domain1.max_interval = 16
kernel.sched_domain.cpu0.domain1.max_newidle_lb_cost = 169622
kernel.sched_domain.cpu0.domain1.min_interval = 8
kernel.sched_domain.cpu0.domain1.name = MC
kernel.sched_domain.cpu0.domain1.newidle_idx = 0
kernel.sched_domain.cpu0.domain1.wake_idx = 0
kernel.sched_domain.cpu1.domain0.busy_factor = 32
kernel.sched_domain.cpu1.domain0.busy_idx = 0
kernel.sched_domain.cpu1.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu1.domain0.flags = 4783
kernel.sched_domain.cpu1.domain0.forkexec_idx = 0
kernel.sched_domain.cpu1.domain0.idle_idx = 0
kernel.sched_domain.cpu1.domain0.imbalance_pct = 110
kernel.sched_domain.cpu1.domain0.max_interval = 4
kernel.sched_domain.cpu1.domain0.max_newidle_lb_cost = 27013
kernel.sched_domain.cpu1.domain0.min_interval = 2
kernel.sched_domain.cpu1.domain0.name = SMT
kernel.sched_domain.cpu1.domain0.newidle_idx = 0
kernel.sched_domain.cpu1.domain0.wake_idx = 0
kernel.sched_domain.cpu1.domain1.busy_factor = 32
kernel.sched_domain.cpu1.domain1.busy_idx = 2
kernel.sched_domain.cpu1.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu1.domain1.flags = 4655
kernel.sched_domain.cpu1.domain1.forkexec_idx = 0
kernel.sched_domain.cpu1.domain1.idle_idx = 0
kernel.sched_domain.cpu1.domain1.imbalance_pct = 117
kernel.sched_domain.cpu1.domain1.max_interval = 16
kernel.sched_domain.cpu1.domain1.max_newidle_lb_cost = 203456
kernel.sched_domain.cpu1.domain1.min_interval = 8
kernel.sched_domain.cpu1.domain1.name = MC
kernel.sched_domain.cpu1.domain1.newidle_idx = 0
kernel.sched_domain.cpu1.domain1.wake_idx = 0
kernel.sched_domain.cpu2.domain0.busy_factor = 32
kernel.sched_domain.cpu2.domain0.busy_idx = 0
kernel.sched_domain.cpu2.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu2.domain0.flags = 4783
kernel.sched_domain.cpu2.domain0.forkexec_idx = 0
kernel.sched_domain.cpu2.domain0.idle_idx = 0
kernel.sched_domain.cpu2.domain0.imbalance_pct = 110
kernel.sched_domain.cpu2.domain0.max_interval = 4
kernel.sched_domain.cpu2.domain0.max_newidle_lb_cost = 57820
kernel.sched_domain.cpu2.domain0.min_interval = 2
kernel.sched_domain.cpu2.domain0.name = SMT
kernel.sched_domain.cpu2.domain0.newidle_idx = 0
kernel.sched_domain.cpu2.domain0.wake_idx = 0
kernel.sched_domain.cpu2.domain1.busy_factor = 32
kernel.sched_domain.cpu2.domain1.busy_idx = 2
kernel.sched_domain.cpu2.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu2.domain1.flags = 4655
kernel.sched_domain.cpu2.domain1.forkexec_idx = 0
kernel.sched_domain.cpu2.domain1.idle_idx = 0
kernel.sched_domain.cpu2.domain1.imbalance_pct = 117
kernel.sched_domain.cpu2.domain1.max_interval = 16
kernel.sched_domain.cpu2.domain1.max_newidle_lb_cost = 142395
kernel.sched_domain.cpu2.domain1.min_interval = 8
kernel.sched_domain.cpu2.domain1.name = MC
kernel.sched_domain.cpu2.domain1.newidle_idx = 0
kernel.sched_domain.cpu2.domain1.wake_idx = 0
kernel.sched_domain.cpu3.domain0.busy_factor = 32
kernel.sched_domain.cpu3.domain0.busy_idx = 0
kernel.sched_domain.cpu3.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu3.domain0.flags = 4783
kernel.sched_domain.cpu3.domain0.forkexec_idx = 0
kernel.sched_domain.cpu3.domain0.idle_idx = 0
kernel.sched_domain.cpu3.domain0.imbalance_pct = 110
kernel.sched_domain.cpu3.domain0.max_interval = 4
kernel.sched_domain.cpu3.domain0.max_newidle_lb_cost = 62576
kernel.sched_domain.cpu3.domain0.min_interval = 2
kernel.sched_domain.cpu3.domain0.name = SMT
kernel.sched_domain.cpu3.domain0.newidle_idx = 0
kernel.sched_domain.cpu3.domain0.wake_idx = 0
kernel.sched_domain.cpu3.domain1.busy_factor = 32
kernel.sched_domain.cpu3.domain1.busy_idx = 2
kernel.sched_domain.cpu3.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu3.domain1.flags = 4655
kernel.sched_domain.cpu3.domain1.forkexec_idx = 0
kernel.sched_domain.cpu3.domain1.idle_idx = 0
kernel.sched_domain.cpu3.domain1.imbalance_pct = 117
kernel.sched_domain.cpu3.domain1.max_interval = 16
kernel.sched_domain.cpu3.domain1.max_newidle_lb_cost = 107913
kernel.sched_domain.cpu3.domain1.min_interval = 8
kernel.sched_domain.cpu3.domain1.name = MC
kernel.sched_domain.cpu3.domain1.newidle_idx = 0
kernel.sched_domain.cpu3.domain1.wake_idx = 0
kernel.sched_domain.cpu4.domain0.busy_factor = 32
kernel.sched_domain.cpu4.domain0.busy_idx = 0
kernel.sched_domain.cpu4.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu4.domain0.flags = 4783
kernel.sched_domain.cpu4.domain0.forkexec_idx = 0
kernel.sched_domain.cpu4.domain0.idle_idx = 0
kernel.sched_domain.cpu4.domain0.imbalance_pct = 110
kernel.sched_domain.cpu4.domain0.max_interval = 4
kernel.sched_domain.cpu4.domain0.max_newidle_lb_cost = 72524
kernel.sched_domain.cpu4.domain0.min_interval = 2
kernel.sched_domain.cpu4.domain0.name = SMT
kernel.sched_domain.cpu4.domain0.newidle_idx = 0
kernel.sched_domain.cpu4.domain0.wake_idx = 0
kernel.sched_domain.cpu4.domain1.busy_factor = 32
kernel.sched_domain.cpu4.domain1.busy_idx = 2
kernel.sched_domain.cpu4.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu4.domain1.flags = 4655
kernel.sched_domain.cpu4.domain1.forkexec_idx = 0
kernel.sched_domain.cpu4.domain1.idle_idx = 0
kernel.sched_domain.cpu4.domain1.imbalance_pct = 117
kernel.sched_domain.cpu4.domain1.max_interval = 16
kernel.sched_domain.cpu4.domain1.max_newidle_lb_cost = 240795
kernel.sched_domain.cpu4.domain1.min_interval = 8
kernel.sched_domain.cpu4.domain1.name = MC
kernel.sched_domain.cpu4.domain1.newidle_idx = 0
kernel.sched_domain.cpu4.domain1.wake_idx = 0
kernel.sched_domain.cpu5.domain0.busy_factor = 32
kernel.sched_domain.cpu5.domain0.busy_idx = 0
kernel.sched_domain.cpu5.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu5.domain0.flags = 4783
kernel.sched_domain.cpu5.domain0.forkexec_idx = 0
kernel.sched_domain.cpu5.domain0.idle_idx = 0
kernel.sched_domain.cpu5.domain0.imbalance_pct = 110
kernel.sched_domain.cpu5.domain0.max_interval = 4
kernel.sched_domain.cpu5.domain0.max_newidle_lb_cost = 64081
kernel.sched_domain.cpu5.domain0.min_interval = 2
kernel.sched_domain.cpu5.domain0.name = SMT
kernel.sched_domain.cpu5.domain0.newidle_idx = 0
kernel.sched_domain.cpu5.domain0.wake_idx = 0
kernel.sched_domain.cpu5.domain1.busy_factor = 32
kernel.sched_domain.cpu5.domain1.busy_idx = 2
kernel.sched_domain.cpu5.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu5.domain1.flags = 4655
kernel.sched_domain.cpu5.domain1.forkexec_idx = 0
kernel.sched_domain.cpu5.domain1.idle_idx = 0
kernel.sched_domain.cpu5.domain1.imbalance_pct = 117
kernel.sched_domain.cpu5.domain1.max_interval = 16
kernel.sched_domain.cpu5.domain1.max_newidle_lb_cost = 114129
kernel.sched_domain.cpu5.domain1.min_interval = 8
kernel.sched_domain.cpu5.domain1.name = MC
kernel.sched_domain.cpu5.domain1.newidle_idx = 0
kernel.sched_domain.cpu5.domain1.wake_idx = 0
kernel.sched_domain.cpu6.domain0.busy_factor = 32
kernel.sched_domain.cpu6.domain0.busy_idx = 0
kernel.sched_domain.cpu6.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu6.domain0.flags = 4783
kernel.sched_domain.cpu6.domain0.forkexec_idx = 0
kernel.sched_domain.cpu6.domain0.idle_idx = 0
kernel.sched_domain.cpu6.domain0.imbalance_pct = 110
kernel.sched_domain.cpu6.domain0.max_interval = 4
kernel.sched_domain.cpu6.domain0.max_newidle_lb_cost = 71346
kernel.sched_domain.cpu6.domain0.min_interval = 2
kernel.sched_domain.cpu6.domain0.name = SMT
kernel.sched_domain.cpu6.domain0.newidle_idx = 0
kernel.sched_domain.cpu6.domain0.wake_idx = 0
kernel.sched_domain.cpu6.domain1.busy_factor = 32
kernel.sched_domain.cpu6.domain1.busy_idx = 2
kernel.sched_domain.cpu6.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu6.domain1.flags = 4655
kernel.sched_domain.cpu6.domain1.forkexec_idx = 0
kernel.sched_domain.cpu6.domain1.idle_idx = 0
kernel.sched_domain.cpu6.domain1.imbalance_pct = 117
kernel.sched_domain.cpu6.domain1.max_interval = 16
kernel.sched_domain.cpu6.domain1.max_newidle_lb_cost = 100683
kernel.sched_domain.cpu6.domain1.min_interval = 8
kernel.sched_domain.cpu6.domain1.name = MC
kernel.sched_domain.cpu6.domain1.newidle_idx = 0
kernel.sched_domain.cpu6.domain1.wake_idx = 0
kernel.sched_domain.cpu7.domain0.busy_factor = 32
kernel.sched_domain.cpu7.domain0.busy_idx = 0
kernel.sched_domain.cpu7.domain0.cache_nice_tries = 0
kernel.sched_domain.cpu7.domain0.flags = 4783
kernel.sched_domain.cpu7.domain0.forkexec_idx = 0
kernel.sched_domain.cpu7.domain0.idle_idx = 0
kernel.sched_domain.cpu7.domain0.imbalance_pct = 110
kernel.sched_domain.cpu7.domain0.max_interval = 4
kernel.sched_domain.cpu7.domain0.max_newidle_lb_cost = 73796
kernel.sched_domain.cpu7.domain0.min_interval = 2
kernel.sched_domain.cpu7.domain0.name = SMT
kernel.sched_domain.cpu7.domain0.newidle_idx = 0
kernel.sched_domain.cpu7.domain0.wake_idx = 0
kernel.sched_domain.cpu7.domain1.busy_factor = 32
kernel.sched_domain.cpu7.domain1.busy_idx = 2
kernel.sched_domain.cpu7.domain1.cache_nice_tries = 1
kernel.sched_domain.cpu7.domain1.flags = 4655
kernel.sched_domain.cpu7.domain1.forkexec_idx = 0
kernel.sched_domain.cpu7.domain1.idle_idx = 0
kernel.sched_domain.cpu7.domain1.imbalance_pct = 117
kernel.sched_domain.cpu7.domain1.max_interval = 16
kernel.sched_domain.cpu7.domain1.max_newidle_lb_cost = 91453
kernel.sched_domain.cpu7.domain1.min_interval = 8
kernel.sched_domain.cpu7.domain1.name = MC
kernel.sched_domain.cpu7.domain1.newidle_idx = 0
kernel.sched_domain.cpu7.domain1.wake_idx = 0
kernel.sched_latency_ns = 24000000
kernel.sched_migration_cost_ns = 500000
kernel.sched_min_granularity_ns = 3000000
kernel.sched_nr_migrate = 32
kernel.sched_rr_timeslice_ms = 100
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_schedstats = 0
kernel.sched_tunable_scaling = 1
kernel.sched_wakeup_granularity_ns = 4000000
kernel.seccomp.actions_avail = kill_process kill_thread trap errno user_notif trace log allow
kernel.seccomp.actions_logged = kill_process kill_thread trap errno user_notif trace log
kernel.sem = 32000	1024000000	500	32000
kernel.sem_next_id = -1
kernel.sg-big-buff = 32768
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399
kernel.shmmni = 4096
kernel.soft_watchdog = 1
kernel.softlockup_all_cpu_backtrace = 0
kernel.softlockup_panic = 0
kernel.stack_tracer_enabled = 0
kernel.sysctl_writes_strict = 1
kernel.sysrq = 176
kernel.tainted = 0
kernel.threads-max = 126672
kernel.timer_migration = 1
kernel.traceoff_on_warning = 0
kernel.tracepoint_printk = 0
kernel.unknown_nmi_panic = 0
kernel.unprivileged_bpf_disabled = 0
kernel.unprivileged_userns_apparmor_policy = 1
kernel.unprivileged_userns_clone = 1
kernel.usermodehelper.bset = 4294967295	63
kernel.usermodehelper.inheritable = 4294967295	63
kernel.version = #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019
kernel.watchdog = 1
kernel.watchdog_cpumask = 0-7
kernel.watchdog_thresh = 10
kernel.yama.ptrace_scope = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0
net.core.bpf_jit_enable = 1
net.core.bpf_jit_harden = 0
net.core.bpf_jit_kallsyms = 0
net.core.bpf_jit_limit = 264241152
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.default_qdisc = fq_codel
net.core.dev_weight = 64
net.core.dev_weight_rx_bias = 1
net.core.dev_weight_tx_bias = 1
net.core.fb_tunnels_only_for_init_net = 0
net.core.flow_limit_cpu_bitmap = 00
net.core.flow_limit_table_len = 4096
net.core.max_skb_frags = 17
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_budget_usecs = 2000
net.core.netdev_max_backlog = 5000
net.core.netdev_rss_key = de:1f:b8:a8:59:5e:9a:09:fb:e6:9a:1d:f5:7e:79:d2:05:a3:65:3c:2f:9e:25:e2:b4:13:02:72:b8:3a:ce:71:65:64:15:99:9c:63:6d:f0:b4:dd:7b:85:cb:3d:7d:6a:54:58:b3:b0
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 20480
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.rps_sock_flow_entries = 0
net.core.somaxconn = 128
net.core.tstamp_allow_data = 1
net.core.warnings = 0
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.cipso_cache_bucket_size = 10
net.ipv4.cipso_cache_enable = 1
net.ipv4.cipso_rbm_optfmt = 0
net.ipv4.cipso_rbm_strictvalid = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.drop_gratuitous_arp = 0
net.ipv4.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.all.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.all.ignore_routes_with_linkdown = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 1
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.route_localnet = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.br-1542969e16e8.accept_local = 0
net.ipv4.conf.br-1542969e16e8.accept_redirects = 1
net.ipv4.conf.br-1542969e16e8.accept_source_route = 1
net.ipv4.conf.br-1542969e16e8.arp_accept = 0
net.ipv4.conf.br-1542969e16e8.arp_announce = 0
net.ipv4.conf.br-1542969e16e8.arp_filter = 0
net.ipv4.conf.br-1542969e16e8.arp_ignore = 0
net.ipv4.conf.br-1542969e16e8.arp_notify = 0
net.ipv4.conf.br-1542969e16e8.bc_forwarding = 0
net.ipv4.conf.br-1542969e16e8.bootp_relay = 0
net.ipv4.conf.br-1542969e16e8.disable_policy = 0
net.ipv4.conf.br-1542969e16e8.disable_xfrm = 0
net.ipv4.conf.br-1542969e16e8.drop_gratuitous_arp = 0
net.ipv4.conf.br-1542969e16e8.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.br-1542969e16e8.force_igmp_version = 0
net.ipv4.conf.br-1542969e16e8.forwarding = 1
net.ipv4.conf.br-1542969e16e8.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.br-1542969e16e8.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.br-1542969e16e8.ignore_routes_with_linkdown = 0
net.ipv4.conf.br-1542969e16e8.log_martians = 0
net.ipv4.conf.br-1542969e16e8.mc_forwarding = 0
net.ipv4.conf.br-1542969e16e8.medium_id = 0
net.ipv4.conf.br-1542969e16e8.promote_secondaries = 0
net.ipv4.conf.br-1542969e16e8.proxy_arp = 0
net.ipv4.conf.br-1542969e16e8.proxy_arp_pvlan = 0
net.ipv4.conf.br-1542969e16e8.route_localnet = 0
net.ipv4.conf.br-1542969e16e8.rp_filter = 1
net.ipv4.conf.br-1542969e16e8.secure_redirects = 1
net.ipv4.conf.br-1542969e16e8.send_redirects = 1
net.ipv4.conf.br-1542969e16e8.shared_media = 1
net.ipv4.conf.br-1542969e16e8.src_valid_mark = 0
net.ipv4.conf.br-1542969e16e8.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.drop_gratuitous_arp = 0
net.ipv4.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.default.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.default.ignore_routes_with_linkdown = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.route_localnet = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.docker0.accept_local = 0
net.ipv4.conf.docker0.accept_redirects = 1
net.ipv4.conf.docker0.accept_source_route = 1
net.ipv4.conf.docker0.arp_accept = 0
net.ipv4.conf.docker0.arp_announce = 0
net.ipv4.conf.docker0.arp_filter = 0
net.ipv4.conf.docker0.arp_ignore = 0
net.ipv4.conf.docker0.arp_notify = 0
net.ipv4.conf.docker0.bc_forwarding = 0
net.ipv4.conf.docker0.bootp_relay = 0
net.ipv4.conf.docker0.disable_policy = 0
net.ipv4.conf.docker0.disable_xfrm = 0
net.ipv4.conf.docker0.drop_gratuitous_arp = 0
net.ipv4.conf.docker0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.docker0.force_igmp_version = 0
net.ipv4.conf.docker0.forwarding = 1
net.ipv4.conf.docker0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.docker0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.docker0.ignore_routes_with_linkdown = 0
net.ipv4.conf.docker0.log_martians = 0
net.ipv4.conf.docker0.mc_forwarding = 0
net.ipv4.conf.docker0.medium_id = 0
net.ipv4.conf.docker0.promote_secondaries = 0
net.ipv4.conf.docker0.proxy_arp = 0
net.ipv4.conf.docker0.proxy_arp_pvlan = 0
net.ipv4.conf.docker0.route_localnet = 0
net.ipv4.conf.docker0.rp_filter = 1
net.ipv4.conf.docker0.secure_redirects = 1
net.ipv4.conf.docker0.send_redirects = 1
net.ipv4.conf.docker0.shared_media = 1
net.ipv4.conf.docker0.src_valid_mark = 0
net.ipv4.conf.docker0.tag = 0
net.ipv4.conf.enp1s0.accept_local = 0
net.ipv4.conf.enp1s0.accept_redirects = 1
net.ipv4.conf.enp1s0.accept_source_route = 1
net.ipv4.conf.enp1s0.arp_accept = 0
net.ipv4.conf.enp1s0.arp_announce = 0
net.ipv4.conf.enp1s0.arp_filter = 0
net.ipv4.conf.enp1s0.arp_ignore = 0
net.ipv4.conf.enp1s0.arp_notify = 0
net.ipv4.conf.enp1s0.bc_forwarding = 0
net.ipv4.conf.enp1s0.bootp_relay = 0
net.ipv4.conf.enp1s0.disable_policy = 0
net.ipv4.conf.enp1s0.disable_xfrm = 0
net.ipv4.conf.enp1s0.drop_gratuitous_arp = 0
net.ipv4.conf.enp1s0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.enp1s0.force_igmp_version = 0
net.ipv4.conf.enp1s0.forwarding = 1
net.ipv4.conf.enp1s0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.enp1s0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.enp1s0.ignore_routes_with_linkdown = 0
net.ipv4.conf.enp1s0.log_martians = 0
net.ipv4.conf.enp1s0.mc_forwarding = 0
net.ipv4.conf.enp1s0.medium_id = 0
net.ipv4.conf.enp1s0.promote_secondaries = 0
net.ipv4.conf.enp1s0.proxy_arp = 0
net.ipv4.conf.enp1s0.proxy_arp_pvlan = 0
net.ipv4.conf.enp1s0.route_localnet = 0
net.ipv4.conf.enp1s0.rp_filter = 1
net.ipv4.conf.enp1s0.secure_redirects = 1
net.ipv4.conf.enp1s0.send_redirects = 1
net.ipv4.conf.enp1s0.shared_media = 1
net.ipv4.conf.enp1s0.src_valid_mark = 0
net.ipv4.conf.enp1s0.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.drop_gratuitous_arp = 0
net.ipv4.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.lo.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.lo.ignore_routes_with_linkdown = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.route_localnet = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.conf.vetheecbe26.accept_local = 0
net.ipv4.conf.vetheecbe26.accept_redirects = 1
net.ipv4.conf.vetheecbe26.accept_source_route = 1
net.ipv4.conf.vetheecbe26.arp_accept = 0
net.ipv4.conf.vetheecbe26.arp_announce = 0
net.ipv4.conf.vetheecbe26.arp_filter = 0
net.ipv4.conf.vetheecbe26.arp_ignore = 0
net.ipv4.conf.vetheecbe26.arp_notify = 0
net.ipv4.conf.vetheecbe26.bc_forwarding = 0
net.ipv4.conf.vetheecbe26.bootp_relay = 0
net.ipv4.conf.vetheecbe26.disable_policy = 0
net.ipv4.conf.vetheecbe26.disable_xfrm = 0
net.ipv4.conf.vetheecbe26.drop_gratuitous_arp = 0
net.ipv4.conf.vetheecbe26.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.vetheecbe26.force_igmp_version = 0
net.ipv4.conf.vetheecbe26.forwarding = 1
net.ipv4.conf.vetheecbe26.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.vetheecbe26.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.vetheecbe26.ignore_routes_with_linkdown = 0
net.ipv4.conf.vetheecbe26.log_martians = 0
net.ipv4.conf.vetheecbe26.mc_forwarding = 0
net.ipv4.conf.vetheecbe26.medium_id = 0
net.ipv4.conf.vetheecbe26.promote_secondaries = 0
net.ipv4.conf.vetheecbe26.proxy_arp = 0
net.ipv4.conf.vetheecbe26.proxy_arp_pvlan = 0
net.ipv4.conf.vetheecbe26.route_localnet = 0
net.ipv4.conf.vetheecbe26.rp_filter = 1
net.ipv4.conf.vetheecbe26.secure_redirects = 1
net.ipv4.conf.vetheecbe26.send_redirects = 1
net.ipv4.conf.vetheecbe26.shared_media = 1
net.ipv4.conf.vetheecbe26.src_valid_mark = 0
net.ipv4.conf.vetheecbe26.tag = 0
net.ipv4.conf.wlp0s20f3.accept_local = 0
net.ipv4.conf.wlp0s20f3.accept_redirects = 1
net.ipv4.conf.wlp0s20f3.accept_source_route = 1
net.ipv4.conf.wlp0s20f3.arp_accept = 0
net.ipv4.conf.wlp0s20f3.arp_announce = 0
net.ipv4.conf.wlp0s20f3.arp_filter = 0
net.ipv4.conf.wlp0s20f3.arp_ignore = 0
net.ipv4.conf.wlp0s20f3.arp_notify = 0
net.ipv4.conf.wlp0s20f3.bc_forwarding = 0
net.ipv4.conf.wlp0s20f3.bootp_relay = 0
net.ipv4.conf.wlp0s20f3.disable_policy = 0
net.ipv4.conf.wlp0s20f3.disable_xfrm = 0
net.ipv4.conf.wlp0s20f3.drop_gratuitous_arp = 0
net.ipv4.conf.wlp0s20f3.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.wlp0s20f3.force_igmp_version = 0
net.ipv4.conf.wlp0s20f3.forwarding = 1
net.ipv4.conf.wlp0s20f3.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.wlp0s20f3.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.wlp0s20f3.ignore_routes_with_linkdown = 0
net.ipv4.conf.wlp0s20f3.log_martians = 0
net.ipv4.conf.wlp0s20f3.mc_forwarding = 0
net.ipv4.conf.wlp0s20f3.medium_id = 0
net.ipv4.conf.wlp0s20f3.promote_secondaries = 0
net.ipv4.conf.wlp0s20f3.proxy_arp = 0
net.ipv4.conf.wlp0s20f3.proxy_arp_pvlan = 0
net.ipv4.conf.wlp0s20f3.route_localnet = 0
net.ipv4.conf.wlp0s20f3.rp_filter = 2
net.ipv4.conf.wlp0s20f3.secure_redirects = 1
net.ipv4.conf.wlp0s20f3.send_redirects = 1
net.ipv4.conf.wlp0s20f3.shared_media = 1
net.ipv4.conf.wlp0s20f3.src_valid_mark = 0
net.ipv4.conf.wlp0s20f3.tag = 0
net.ipv4.fib_multipath_hash_policy = 0
net.ipv4.fib_multipath_use_neigh = 0
net.ipv4.fwmark_reflect = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_msgs_burst = 50
net.ipv4.icmp_msgs_per_sec = 1000
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_link_local_mcast_reports = 1
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_qrv = 2
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_early_demux = 1
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv4.ip_local_port_range = 32768	60999
net.ipv4.ip_local_reserved_ports = 
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ip_unprivileged_port_start = 1024
net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 0
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.br-1542969e16e8.anycast_delay = 100
net.ipv4.neigh.br-1542969e16e8.app_solicit = 0
net.ipv4.neigh.br-1542969e16e8.base_reachable_time_ms = 30000
net.ipv4.neigh.br-1542969e16e8.delay_first_probe_time = 5
net.ipv4.neigh.br-1542969e16e8.gc_stale_time = 60
net.ipv4.neigh.br-1542969e16e8.locktime = 100
net.ipv4.neigh.br-1542969e16e8.mcast_resolicit = 0
net.ipv4.neigh.br-1542969e16e8.mcast_solicit = 3
net.ipv4.neigh.br-1542969e16e8.proxy_delay = 80
net.ipv4.neigh.br-1542969e16e8.proxy_qlen = 64
net.ipv4.neigh.br-1542969e16e8.retrans_time_ms = 1000
net.ipv4.neigh.br-1542969e16e8.ucast_solicit = 3
net.ipv4.neigh.br-1542969e16e8.unres_qlen = 101
net.ipv4.neigh.br-1542969e16e8.unres_qlen_bytes = 212992
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_resolicit = 0
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 101
net.ipv4.neigh.default.unres_qlen_bytes = 212992
net.ipv4.neigh.docker0.anycast_delay = 100
net.ipv4.neigh.docker0.app_solicit = 0
net.ipv4.neigh.docker0.base_reachable_time_ms = 30000
net.ipv4.neigh.docker0.delay_first_probe_time = 5
net.ipv4.neigh.docker0.gc_stale_time = 60
net.ipv4.neigh.docker0.locktime = 100
net.ipv4.neigh.docker0.mcast_resolicit = 0
net.ipv4.neigh.docker0.mcast_solicit = 3
net.ipv4.neigh.docker0.proxy_delay = 80
net.ipv4.neigh.docker0.proxy_qlen = 64
net.ipv4.neigh.docker0.retrans_time_ms = 1000
net.ipv4.neigh.docker0.ucast_solicit = 3
net.ipv4.neigh.docker0.unres_qlen = 101
net.ipv4.neigh.docker0.unres_qlen_bytes = 212992
net.ipv4.neigh.enp1s0.anycast_delay = 100
net.ipv4.neigh.enp1s0.app_solicit = 0
net.ipv4.neigh.enp1s0.base_reachable_time_ms = 30000
net.ipv4.neigh.enp1s0.delay_first_probe_time = 5
net.ipv4.neigh.enp1s0.gc_stale_time = 60
net.ipv4.neigh.enp1s0.locktime = 100
net.ipv4.neigh.enp1s0.mcast_resolicit = 0
net.ipv4.neigh.enp1s0.mcast_solicit = 3
net.ipv4.neigh.enp1s0.proxy_delay = 80
net.ipv4.neigh.enp1s0.proxy_qlen = 64
net.ipv4.neigh.enp1s0.retrans_time_ms = 1000
net.ipv4.neigh.enp1s0.ucast_solicit = 3
net.ipv4.neigh.enp1s0.unres_qlen = 101
net.ipv4.neigh.enp1s0.unres_qlen_bytes = 212992
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_resolicit = 0
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 101
net.ipv4.neigh.lo.unres_qlen_bytes = 212992
net.ipv4.neigh.vetheecbe26.anycast_delay = 100
net.ipv4.neigh.vetheecbe26.app_solicit = 0
net.ipv4.neigh.vetheecbe26.base_reachable_time_ms = 30000
net.ipv4.neigh.vetheecbe26.delay_first_probe_time = 5
net.ipv4.neigh.vetheecbe26.gc_stale_time = 60
net.ipv4.neigh.vetheecbe26.locktime = 100
net.ipv4.neigh.vetheecbe26.mcast_resolicit = 0
net.ipv4.neigh.vetheecbe26.mcast_solicit = 3
net.ipv4.neigh.vetheecbe26.proxy_delay = 80
net.ipv4.neigh.vetheecbe26.proxy_qlen = 64
net.ipv4.neigh.vetheecbe26.retrans_time_ms = 1000
net.ipv4.neigh.vetheecbe26.ucast_solicit = 3
net.ipv4.neigh.vetheecbe26.unres_qlen = 101
net.ipv4.neigh.vetheecbe26.unres_qlen_bytes = 212992
net.ipv4.neigh.wlp0s20f3.anycast_delay = 100
net.ipv4.neigh.wlp0s20f3.app_solicit = 0
net.ipv4.neigh.wlp0s20f3.base_reachable_time_ms = 30000
net.ipv4.neigh.wlp0s20f3.delay_first_probe_time = 5
net.ipv4.neigh.wlp0s20f3.gc_stale_time = 60
net.ipv4.neigh.wlp0s20f3.locktime = 100
net.ipv4.neigh.wlp0s20f3.mcast_resolicit = 0
net.ipv4.neigh.wlp0s20f3.mcast_solicit = 3
net.ipv4.neigh.wlp0s20f3.proxy_delay = 80
net.ipv4.neigh.wlp0s20f3.proxy_qlen = 64
net.ipv4.neigh.wlp0s20f3.retrans_time_ms = 1000
net.ipv4.neigh.wlp0s20f3.ucast_solicit = 3
net.ipv4.neigh.wlp0s20f3.unres_qlen = 101
net.ipv4.neigh.wlp0s20f3.unres_qlen_bytes = 212992
net.ipv4.ping_group_range = 1	0
net.ipv4.raw_l3mdev_accept = 1
net.ipv4.route.error_burst = 1250
net.ipv4.route.error_cost = 250
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = -1
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 2147483647
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 5
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 5120
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno cubic
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = reno cubic
net.ipv4.tcp_available_ulp = 
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_comp_sack_delay_ns = 1000000
net.ipv4.tcp_comp_sack_nr = 44
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 0
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 1048576
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.tcp_max_tw_buckets = 65536
net.ipv4.tcp_mem = 16384000	16384000	16384000
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_snd_mss = 48
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 10240	87380	16777216
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 10240	87380	16777216
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_early_demux = 1
net.ipv4.udp_l3mdev_accept = 0
net.ipv4.udp_mem = 377640	503523	755280
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.xfrm4_gc_thresh = 32768
net.ipv6.anycast_src_echo_reply = 0
net.ipv6.auto_flowlabels = 1
net.ipv6.bindv6only = 0
net.ipv6.calipso_cache_bucket_size = 10
net.ipv6.calipso_cache_enable = 1
net.ipv6.conf.all.accept_dad = 0
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_from_local = 0
net.ipv6.conf.all.accept_ra_min_hop_limit = 1
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.all.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.addr_gen_mode = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.disable_policy = 0
net.ipv6.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.all.drop_unsolicited_na = 0
net.ipv6.conf.all.enhanced_dad = 1
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.ignore_routes_with_linkdown = 0
net.ipv6.conf.all.keep_addr_on_down = 0
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.ndisc_tclass = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitation_max_interval = 3600
net.ipv6.conf.all.router_solicitations = -1
net.ipv6.conf.all.seg6_enabled = 0
net.ipv6.conf.all.seg6_require_hmac = 0
net.ipv6.conf.all.suppress_frag_ndisc = 1
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_oif_addrs_only = 0
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.br-1542969e16e8.accept_dad = 1
net.ipv6.conf.br-1542969e16e8.accept_ra = 1
net.ipv6.conf.br-1542969e16e8.accept_ra_defrtr = 1
net.ipv6.conf.br-1542969e16e8.accept_ra_from_local = 0
net.ipv6.conf.br-1542969e16e8.accept_ra_min_hop_limit = 1
net.ipv6.conf.br-1542969e16e8.accept_ra_mtu = 1
net.ipv6.conf.br-1542969e16e8.accept_ra_pinfo = 1
net.ipv6.conf.br-1542969e16e8.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.br-1542969e16e8.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.br-1542969e16e8.accept_ra_rtr_pref = 1
net.ipv6.conf.br-1542969e16e8.accept_redirects = 1
net.ipv6.conf.br-1542969e16e8.accept_source_route = 0
net.ipv6.conf.br-1542969e16e8.addr_gen_mode = 0
net.ipv6.conf.br-1542969e16e8.autoconf = 1
net.ipv6.conf.br-1542969e16e8.dad_transmits = 1
net.ipv6.conf.br-1542969e16e8.disable_ipv6 = 0
net.ipv6.conf.br-1542969e16e8.disable_policy = 0
net.ipv6.conf.br-1542969e16e8.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.br-1542969e16e8.drop_unsolicited_na = 0
net.ipv6.conf.br-1542969e16e8.enhanced_dad = 1
net.ipv6.conf.br-1542969e16e8.force_mld_version = 0
net.ipv6.conf.br-1542969e16e8.force_tllao = 0
net.ipv6.conf.br-1542969e16e8.forwarding = 0
net.ipv6.conf.br-1542969e16e8.hop_limit = 64
net.ipv6.conf.br-1542969e16e8.ignore_routes_with_linkdown = 0
net.ipv6.conf.br-1542969e16e8.keep_addr_on_down = 0
net.ipv6.conf.br-1542969e16e8.max_addresses = 16
net.ipv6.conf.br-1542969e16e8.max_desync_factor = 600
net.ipv6.conf.br-1542969e16e8.mc_forwarding = 0
net.ipv6.conf.br-1542969e16e8.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.br-1542969e16e8.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.br-1542969e16e8.mtu = 1500
net.ipv6.conf.br-1542969e16e8.ndisc_notify = 0
net.ipv6.conf.br-1542969e16e8.ndisc_tclass = 0
net.ipv6.conf.br-1542969e16e8.proxy_ndp = 0
net.ipv6.conf.br-1542969e16e8.regen_max_retry = 3
net.ipv6.conf.br-1542969e16e8.router_probe_interval = 60
net.ipv6.conf.br-1542969e16e8.router_solicitation_delay = 1
net.ipv6.conf.br-1542969e16e8.router_solicitation_interval = 4
net.ipv6.conf.br-1542969e16e8.router_solicitation_max_interval = 3600
net.ipv6.conf.br-1542969e16e8.router_solicitations = -1
net.ipv6.conf.br-1542969e16e8.seg6_enabled = 0
net.ipv6.conf.br-1542969e16e8.seg6_require_hmac = 0
net.ipv6.conf.br-1542969e16e8.suppress_frag_ndisc = 1
net.ipv6.conf.br-1542969e16e8.temp_prefered_lft = 86400
net.ipv6.conf.br-1542969e16e8.temp_valid_lft = 604800
net.ipv6.conf.br-1542969e16e8.use_oif_addrs_only = 0
net.ipv6.conf.br-1542969e16e8.use_tempaddr = 2
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_from_local = 0
net.ipv6.conf.default.accept_ra_min_hop_limit = 1
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.default.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.addr_gen_mode = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.disable_policy = 0
net.ipv6.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.default.drop_unsolicited_na = 0
net.ipv6.conf.default.enhanced_dad = 1
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.ignore_routes_with_linkdown = 0
net.ipv6.conf.default.keep_addr_on_down = 0
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.ndisc_tclass = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitation_max_interval = 3600
net.ipv6.conf.default.router_solicitations = -1
net.ipv6.conf.default.seg6_enabled = 0
net.ipv6.conf.default.seg6_require_hmac = 0
net.ipv6.conf.default.suppress_frag_ndisc = 1
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_oif_addrs_only = 0
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.docker0.accept_dad = 1
net.ipv6.conf.docker0.accept_ra = 1
net.ipv6.conf.docker0.accept_ra_defrtr = 1
net.ipv6.conf.docker0.accept_ra_from_local = 0
net.ipv6.conf.docker0.accept_ra_min_hop_limit = 1
net.ipv6.conf.docker0.accept_ra_mtu = 1
net.ipv6.conf.docker0.accept_ra_pinfo = 1
net.ipv6.conf.docker0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.docker0.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.docker0.accept_ra_rtr_pref = 1
net.ipv6.conf.docker0.accept_redirects = 1
net.ipv6.conf.docker0.accept_source_route = 0
net.ipv6.conf.docker0.addr_gen_mode = 0
net.ipv6.conf.docker0.autoconf = 1
net.ipv6.conf.docker0.dad_transmits = 1
net.ipv6.conf.docker0.disable_ipv6 = 0
net.ipv6.conf.docker0.disable_policy = 0
net.ipv6.conf.docker0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.docker0.drop_unsolicited_na = 0
net.ipv6.conf.docker0.enhanced_dad = 1
net.ipv6.conf.docker0.force_mld_version = 0
net.ipv6.conf.docker0.force_tllao = 0
net.ipv6.conf.docker0.forwarding = 0
net.ipv6.conf.docker0.hop_limit = 64
net.ipv6.conf.docker0.ignore_routes_with_linkdown = 0
net.ipv6.conf.docker0.keep_addr_on_down = 0
net.ipv6.conf.docker0.max_addresses = 16
net.ipv6.conf.docker0.max_desync_factor = 600
net.ipv6.conf.docker0.mc_forwarding = 0
net.ipv6.conf.docker0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.docker0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.docker0.mtu = 1500
net.ipv6.conf.docker0.ndisc_notify = 0
net.ipv6.conf.docker0.ndisc_tclass = 0
net.ipv6.conf.docker0.proxy_ndp = 0
net.ipv6.conf.docker0.regen_max_retry = 3
net.ipv6.conf.docker0.router_probe_interval = 60
net.ipv6.conf.docker0.router_solicitation_delay = 1
net.ipv6.conf.docker0.router_solicitation_interval = 4
net.ipv6.conf.docker0.router_solicitation_max_interval = 3600
net.ipv6.conf.docker0.router_solicitations = -1
net.ipv6.conf.docker0.seg6_enabled = 0
net.ipv6.conf.docker0.seg6_require_hmac = 0
net.ipv6.conf.docker0.suppress_frag_ndisc = 1
net.ipv6.conf.docker0.temp_prefered_lft = 86400
net.ipv6.conf.docker0.temp_valid_lft = 604800
net.ipv6.conf.docker0.use_oif_addrs_only = 0
net.ipv6.conf.docker0.use_tempaddr = 2
net.ipv6.conf.enp1s0.accept_dad = 1
net.ipv6.conf.enp1s0.accept_ra = 0
net.ipv6.conf.enp1s0.accept_ra_defrtr = 0
net.ipv6.conf.enp1s0.accept_ra_from_local = 0
net.ipv6.conf.enp1s0.accept_ra_min_hop_limit = 1
net.ipv6.conf.enp1s0.accept_ra_mtu = 1
net.ipv6.conf.enp1s0.accept_ra_pinfo = 0
net.ipv6.conf.enp1s0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp1s0.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.enp1s0.accept_ra_rtr_pref = 0
net.ipv6.conf.enp1s0.accept_redirects = 1
net.ipv6.conf.enp1s0.accept_source_route = 0
net.ipv6.conf.enp1s0.addr_gen_mode = 1
net.ipv6.conf.enp1s0.autoconf = 1
net.ipv6.conf.enp1s0.dad_transmits = 1
net.ipv6.conf.enp1s0.disable_ipv6 = 0
net.ipv6.conf.enp1s0.disable_policy = 0
net.ipv6.conf.enp1s0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.enp1s0.drop_unsolicited_na = 0
net.ipv6.conf.enp1s0.enhanced_dad = 1
net.ipv6.conf.enp1s0.force_mld_version = 0
net.ipv6.conf.enp1s0.force_tllao = 0
net.ipv6.conf.enp1s0.forwarding = 0
net.ipv6.conf.enp1s0.hop_limit = 64
net.ipv6.conf.enp1s0.ignore_routes_with_linkdown = 0
net.ipv6.conf.enp1s0.keep_addr_on_down = 0
net.ipv6.conf.enp1s0.max_addresses = 16
net.ipv6.conf.enp1s0.max_desync_factor = 600
net.ipv6.conf.enp1s0.mc_forwarding = 0
net.ipv6.conf.enp1s0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp1s0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp1s0.mtu = 1500
net.ipv6.conf.enp1s0.ndisc_notify = 0
net.ipv6.conf.enp1s0.ndisc_tclass = 0
net.ipv6.conf.enp1s0.proxy_ndp = 0
net.ipv6.conf.enp1s0.regen_max_retry = 3
net.ipv6.conf.enp1s0.router_probe_interval = 60
net.ipv6.conf.enp1s0.router_solicitation_delay = 1
net.ipv6.conf.enp1s0.router_solicitation_interval = 4
net.ipv6.conf.enp1s0.router_solicitation_max_interval = 3600
net.ipv6.conf.enp1s0.router_solicitations = -1
net.ipv6.conf.enp1s0.seg6_enabled = 0
net.ipv6.conf.enp1s0.seg6_require_hmac = 0
net.ipv6.conf.enp1s0.suppress_frag_ndisc = 1
net.ipv6.conf.enp1s0.temp_prefered_lft = 86400
net.ipv6.conf.enp1s0.temp_valid_lft = 604800
net.ipv6.conf.enp1s0.use_oif_addrs_only = 0
net.ipv6.conf.enp1s0.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_from_local = 0
net.ipv6.conf.lo.accept_ra_min_hop_limit = 1
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.lo.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.addr_gen_mode = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.disable_policy = 0
net.ipv6.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.lo.drop_unsolicited_na = 0
net.ipv6.conf.lo.enhanced_dad = 1
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
net.ipv6.conf.lo.keep_addr_on_down = 0
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.ndisc_tclass = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitation_max_interval = 3600
net.ipv6.conf.lo.router_solicitations = -1
net.ipv6.conf.lo.seg6_enabled = 0
net.ipv6.conf.lo.seg6_require_hmac = 0
net.ipv6.conf.lo.suppress_frag_ndisc = 1
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_oif_addrs_only = 0
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.conf.vetheecbe26.accept_dad = 1
net.ipv6.conf.vetheecbe26.accept_ra = 1
net.ipv6.conf.vetheecbe26.accept_ra_defrtr = 1
net.ipv6.conf.vetheecbe26.accept_ra_from_local = 0
net.ipv6.conf.vetheecbe26.accept_ra_min_hop_limit = 1
net.ipv6.conf.vetheecbe26.accept_ra_mtu = 1
net.ipv6.conf.vetheecbe26.accept_ra_pinfo = 1
net.ipv6.conf.vetheecbe26.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.vetheecbe26.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.vetheecbe26.accept_ra_rtr_pref = 1
net.ipv6.conf.vetheecbe26.accept_redirects = 1
net.ipv6.conf.vetheecbe26.accept_source_route = 0
net.ipv6.conf.vetheecbe26.addr_gen_mode = 0
net.ipv6.conf.vetheecbe26.autoconf = 1
net.ipv6.conf.vetheecbe26.dad_transmits = 1
net.ipv6.conf.vetheecbe26.disable_ipv6 = 0
net.ipv6.conf.vetheecbe26.disable_policy = 0
net.ipv6.conf.vetheecbe26.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.vetheecbe26.drop_unsolicited_na = 0
net.ipv6.conf.vetheecbe26.enhanced_dad = 1
net.ipv6.conf.vetheecbe26.force_mld_version = 0
net.ipv6.conf.vetheecbe26.force_tllao = 0
net.ipv6.conf.vetheecbe26.forwarding = 0
net.ipv6.conf.vetheecbe26.hop_limit = 64
net.ipv6.conf.vetheecbe26.ignore_routes_with_linkdown = 0
net.ipv6.conf.vetheecbe26.keep_addr_on_down = 0
net.ipv6.conf.vetheecbe26.max_addresses = 16
net.ipv6.conf.vetheecbe26.max_desync_factor = 600
net.ipv6.conf.vetheecbe26.mc_forwarding = 0
net.ipv6.conf.vetheecbe26.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.vetheecbe26.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.vetheecbe26.mtu = 1500
net.ipv6.conf.vetheecbe26.ndisc_notify = 0
net.ipv6.conf.vetheecbe26.ndisc_tclass = 0
net.ipv6.conf.vetheecbe26.proxy_ndp = 0
net.ipv6.conf.vetheecbe26.regen_max_retry = 3
net.ipv6.conf.vetheecbe26.router_probe_interval = 60
net.ipv6.conf.vetheecbe26.router_solicitation_delay = 1
net.ipv6.conf.vetheecbe26.router_solicitation_interval = 4
net.ipv6.conf.vetheecbe26.router_solicitation_max_interval = 3600
net.ipv6.conf.vetheecbe26.router_solicitations = -1
net.ipv6.conf.vetheecbe26.seg6_enabled = 0
net.ipv6.conf.vetheecbe26.seg6_require_hmac = 0
net.ipv6.conf.vetheecbe26.suppress_frag_ndisc = 1
net.ipv6.conf.vetheecbe26.temp_prefered_lft = 86400
net.ipv6.conf.vetheecbe26.temp_valid_lft = 604800
net.ipv6.conf.vetheecbe26.use_oif_addrs_only = 0
net.ipv6.conf.vetheecbe26.use_tempaddr = 2
net.ipv6.conf.wlp0s20f3.accept_dad = 1
net.ipv6.conf.wlp0s20f3.accept_ra = 1
net.ipv6.conf.wlp0s20f3.accept_ra_defrtr = 0
net.ipv6.conf.wlp0s20f3.accept_ra_from_local = 0
net.ipv6.conf.wlp0s20f3.accept_ra_min_hop_limit = 1
net.ipv6.conf.wlp0s20f3.accept_ra_mtu = 1
net.ipv6.conf.wlp0s20f3.accept_ra_pinfo = 0
net.ipv6.conf.wlp0s20f3.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.wlp0s20f3.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.wlp0s20f3.accept_ra_rtr_pref = 0
net.ipv6.conf.wlp0s20f3.accept_redirects = 1
net.ipv6.conf.wlp0s20f3.accept_source_route = 0
net.ipv6.conf.wlp0s20f3.addr_gen_mode = 1
net.ipv6.conf.wlp0s20f3.autoconf = 1
net.ipv6.conf.wlp0s20f3.dad_transmits = 1
net.ipv6.conf.wlp0s20f3.disable_ipv6 = 0
net.ipv6.conf.wlp0s20f3.disable_policy = 0
net.ipv6.conf.wlp0s20f3.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.wlp0s20f3.drop_unsolicited_na = 0
net.ipv6.conf.wlp0s20f3.enhanced_dad = 1
net.ipv6.conf.wlp0s20f3.force_mld_version = 0
net.ipv6.conf.wlp0s20f3.force_tllao = 0
net.ipv6.conf.wlp0s20f3.forwarding = 0
net.ipv6.conf.wlp0s20f3.hop_limit = 64
net.ipv6.conf.wlp0s20f3.ignore_routes_with_linkdown = 0
net.ipv6.conf.wlp0s20f3.keep_addr_on_down = 0
net.ipv6.conf.wlp0s20f3.max_addresses = 16
net.ipv6.conf.wlp0s20f3.max_desync_factor = 600
net.ipv6.conf.wlp0s20f3.mc_forwarding = 0
net.ipv6.conf.wlp0s20f3.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.wlp0s20f3.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.wlp0s20f3.mtu = 1500
net.ipv6.conf.wlp0s20f3.ndisc_notify = 0
net.ipv6.conf.wlp0s20f3.ndisc_tclass = 0
net.ipv6.conf.wlp0s20f3.proxy_ndp = 0
net.ipv6.conf.wlp0s20f3.regen_max_retry = 3
net.ipv6.conf.wlp0s20f3.router_probe_interval = 60
net.ipv6.conf.wlp0s20f3.router_solicitation_delay = 1
net.ipv6.conf.wlp0s20f3.router_solicitation_interval = 4
net.ipv6.conf.wlp0s20f3.router_solicitation_max_interval = 3600
net.ipv6.conf.wlp0s20f3.router_solicitations = -1
net.ipv6.conf.wlp0s20f3.seg6_enabled = 0
net.ipv6.conf.wlp0s20f3.seg6_require_hmac = 0
net.ipv6.conf.wlp0s20f3.suppress_frag_ndisc = 1
net.ipv6.conf.wlp0s20f3.temp_prefered_lft = 86400
net.ipv6.conf.wlp0s20f3.temp_valid_lft = 604800
net.ipv6.conf.wlp0s20f3.use_oif_addrs_only = 0
net.ipv6.conf.wlp0s20f3.use_tempaddr = 2
net.ipv6.fib_multipath_hash_policy = 0
net.ipv6.flowlabel_consistency = 1
net.ipv6.flowlabel_reflect = 0
net.ipv6.flowlabel_state_ranges = 0
net.ipv6.fwmark_reflect = 0
net.ipv6.icmp.echo_ignore_all = 0
net.ipv6.icmp.ratelimit = 1000
net.ipv6.idgen_delay = 1
net.ipv6.idgen_retries = 3
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 0
net.ipv6.ip6frag_time = 60
net.ipv6.ip_nonlocal_bind = 0
net.ipv6.max_dst_opts_length = 2147483647
net.ipv6.max_dst_opts_number = 8
net.ipv6.max_hbh_length = 2147483647
net.ipv6.max_hbh_opts_number = 8
net.ipv6.mld_max_msf = 64
net.ipv6.mld_qrv = 2
net.ipv6.neigh.br-1542969e16e8.anycast_delay = 100
net.ipv6.neigh.br-1542969e16e8.app_solicit = 0
net.ipv6.neigh.br-1542969e16e8.base_reachable_time_ms = 30000
net.ipv6.neigh.br-1542969e16e8.delay_first_probe_time = 5
net.ipv6.neigh.br-1542969e16e8.gc_stale_time = 60
net.ipv6.neigh.br-1542969e16e8.locktime = 0
net.ipv6.neigh.br-1542969e16e8.mcast_resolicit = 0
net.ipv6.neigh.br-1542969e16e8.mcast_solicit = 3
net.ipv6.neigh.br-1542969e16e8.proxy_delay = 80
net.ipv6.neigh.br-1542969e16e8.proxy_qlen = 64
net.ipv6.neigh.br-1542969e16e8.retrans_time_ms = 1000
net.ipv6.neigh.br-1542969e16e8.ucast_solicit = 3
net.ipv6.neigh.br-1542969e16e8.unres_qlen = 101
net.ipv6.neigh.br-1542969e16e8.unres_qlen_bytes = 212992
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_resolicit = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 101
net.ipv6.neigh.default.unres_qlen_bytes = 212992
net.ipv6.neigh.docker0.anycast_delay = 100
net.ipv6.neigh.docker0.app_solicit = 0
net.ipv6.neigh.docker0.base_reachable_time_ms = 30000
net.ipv6.neigh.docker0.delay_first_probe_time = 5
net.ipv6.neigh.docker0.gc_stale_time = 60
net.ipv6.neigh.docker0.locktime = 0
net.ipv6.neigh.docker0.mcast_resolicit = 0
net.ipv6.neigh.docker0.mcast_solicit = 3
net.ipv6.neigh.docker0.proxy_delay = 80
net.ipv6.neigh.docker0.proxy_qlen = 64
net.ipv6.neigh.docker0.retrans_time_ms = 1000
net.ipv6.neigh.docker0.ucast_solicit = 3
net.ipv6.neigh.docker0.unres_qlen = 101
net.ipv6.neigh.docker0.unres_qlen_bytes = 212992
net.ipv6.neigh.enp1s0.anycast_delay = 100
net.ipv6.neigh.enp1s0.app_solicit = 0
net.ipv6.neigh.enp1s0.base_reachable_time_ms = 30000
net.ipv6.neigh.enp1s0.delay_first_probe_time = 5
net.ipv6.neigh.enp1s0.gc_stale_time = 60
net.ipv6.neigh.enp1s0.locktime = 0
net.ipv6.neigh.enp1s0.mcast_resolicit = 0
net.ipv6.neigh.enp1s0.mcast_solicit = 3
net.ipv6.neigh.enp1s0.proxy_delay = 80
net.ipv6.neigh.enp1s0.proxy_qlen = 64
net.ipv6.neigh.enp1s0.retrans_time_ms = 1000
net.ipv6.neigh.enp1s0.ucast_solicit = 3
net.ipv6.neigh.enp1s0.unres_qlen = 101
net.ipv6.neigh.enp1s0.unres_qlen_bytes = 212992
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_resolicit = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 101
net.ipv6.neigh.lo.unres_qlen_bytes = 212992
net.ipv6.neigh.vetheecbe26.anycast_delay = 100
net.ipv6.neigh.vetheecbe26.app_solicit = 0
net.ipv6.neigh.vetheecbe26.base_reachable_time_ms = 30000
net.ipv6.neigh.vetheecbe26.delay_first_probe_time = 5
net.ipv6.neigh.vetheecbe26.gc_stale_time = 60
net.ipv6.neigh.vetheecbe26.locktime = 0
net.ipv6.neigh.vetheecbe26.mcast_resolicit = 0
net.ipv6.neigh.vetheecbe26.mcast_solicit = 3
net.ipv6.neigh.vetheecbe26.proxy_delay = 80
net.ipv6.neigh.vetheecbe26.proxy_qlen = 64
net.ipv6.neigh.vetheecbe26.retrans_time_ms = 1000
net.ipv6.neigh.vetheecbe26.ucast_solicit = 3
net.ipv6.neigh.vetheecbe26.unres_qlen = 101
net.ipv6.neigh.vetheecbe26.unres_qlen_bytes = 212992
net.ipv6.neigh.wlp0s20f3.anycast_delay = 100
net.ipv6.neigh.wlp0s20f3.app_solicit = 0
net.ipv6.neigh.wlp0s20f3.base_reachable_time_ms = 30000
net.ipv6.neigh.wlp0s20f3.delay_first_probe_time = 5
net.ipv6.neigh.wlp0s20f3.gc_stale_time = 60
net.ipv6.neigh.wlp0s20f3.locktime = 0
net.ipv6.neigh.wlp0s20f3.mcast_resolicit = 0
net.ipv6.neigh.wlp0s20f3.mcast_solicit = 3
net.ipv6.neigh.wlp0s20f3.proxy_delay = 80
net.ipv6.neigh.wlp0s20f3.proxy_qlen = 64
net.ipv6.neigh.wlp0s20f3.retrans_time_ms = 1000
net.ipv6.neigh.wlp0s20f3.ucast_solicit = 3
net.ipv6.neigh.wlp0s20f3.unres_qlen = 101
net.ipv6.neigh.wlp0s20f3.unres_qlen_bytes = 212992
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.route.skip_notify_on_dev_down = 0
net.ipv6.seg6_flowlabel = 0
net.ipv6.xfrm6_gc_thresh = 32768
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 65536
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 75
net.netfilter.nf_conntrack_dccp_loose = 1
net.netfilter.nf_conntrack_dccp_timeout_closereq = 64
net.netfilter.nf_conntrack_dccp_timeout_closing = 64
net.netfilter.nf_conntrack_dccp_timeout_open = 43200
net.netfilter.nf_conntrack_dccp_timeout_partopen = 480
net.netfilter.nf_conntrack_dccp_timeout_request = 240
net.netfilter.nf_conntrack_dccp_timeout_respond = 480
net.netfilter.nf_conntrack_dccp_timeout_timewait = 240
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 1024
net.netfilter.nf_conntrack_frag6_high_thresh = 4194304
net.netfilter.nf_conntrack_frag6_low_thresh = 3145728
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 0
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_sctp_timeout_closed = 10
net.netfilter.nf_conntrack_sctp_timeout_cookie_echoed = 3
net.netfilter.nf_conntrack_sctp_timeout_cookie_wait = 3
net.netfilter.nf_conntrack_sctp_timeout_established = 432000
net.netfilter.nf_conntrack_sctp_timeout_heartbeat_acked = 210
net.netfilter.nf_conntrack_sctp_timeout_heartbeat_sent = 30
net.netfilter.nf_conntrack_sctp_timeout_shutdown_ack_sent = 3
net.netfilter.nf_conntrack_sctp_timeout_shutdown_recd = 0
net.netfilter.nf_conntrack_sctp_timeout_shutdown_sent = 0
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_timestamp = 0
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 120
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = NONE
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = NONE
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.netfilter.nf_log_all_netns = 0
net.nf_conntrack_max = 262144
net.unix.max_dgram_qlen = 512
user.max_cgroup_namespaces = 63336
user.max_inotify_instances = 128
user.max_inotify_watches = 524288
user.max_ipc_namespaces = 63336
user.max_mnt_namespaces = 63336
user.max_net_namespaces = 63336
user.max_pid_namespaces = 63336
user.max_user_namespaces = 63336
user.max_uts_namespaces = 63336
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.compact_unevictable_allowed = 1
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256	256	32	0	0
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 67584
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.mmap_rnd_bits = 28
vm.mmap_rnd_compat_bits = 8
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.numa_stat = 1
vm.numa_zonelist_order = Node
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_kbytes = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.watermark_boost_factor = 15000
vm.watermark_scale_factor = 10
vm.zone_reclaim_mode = 0

Dirty code sample to reproduce the problem: https://gist.github.com/R-Romanov/f9c0f575670b4d078916ae8d98a1243b

@erikdubbelboer,

client.DoTimeout() return: "timeout"

I am working with Alexa top 1 million sites (http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
Also tested on several http bin servers on the local network, results are very similar.

Using request.SetConnectionClose () did not help :(

erikdubbelboer · 2019-09-22T13:00:09Z

I'm trying to reproduce your issue with your code. The first thing I notice is that your url variable is shared between all your goroutines which means a lot of them will be requesting the wrong url as the variable will already have been overwritten with the next one.

The issue I'm running into right now is that my DNS, 8.8.8.8, is returning a lot of no such host because I'm doing too many lookups per second. How are you handling this?

scanpatch · 2019-09-22T19:48:17Z

@erikdubbelboer,
My ISP does not limit requests to its name server.

What can be done for the test environment only:

Add more nameservers to /etc/resolv.conf (may increase query execution time)
Do not work with Alexa top 1 million sites, but use the httpbin server on the local network and send all requests to it

In production I plan to use my own name server without any restrictions.

erikdubbelboer · 2019-09-24T14:30:35Z

I resolved a bunch of domains and saved them to a file to use so the DNS isn't an issue. I also added some test code: https://gist.github.com/erikdubbelboer/581e825962b14792d59b23b1de34bda4#file-test-go
With this code I'm getting the timeout problems as soon as it's doing too many requests at the same time. But I'm seeing the exact same problem with net/http. So this seems to be a Go issue where it can't handle too many concurrent connections. I'll need to do a lot more investigating to see if I can find the issue in Go.

erikdubbelboer · 2019-09-25T11:23:44Z

I have created a simple program that just sends and receives packets and measures the average latency: https://gist.github.com/erikdubbelboer/7b8f2f5af796ebacb087fb866e9b8dc2

As you can see it opens 400 more connections every 4 seconds. The server side responds after 200 milliseconds.

Below is the output when I run this program. At it's max the program uses less than 40% of my CPU. But as you can see the average latency keeps increasing. This shows that Go has trouble handling a lot of IO events efficiently. In the end it even returns EOF which should not be possible.

$ go run test3.go 
 400: 200.664389ms (1299)
 400: 200.874264ms (1991)
 400: 201.097414ms (1999)
 400: 201.449586ms (1986)
 800: 201.063217ms (2342)
 800: 201.073388ms (3988)
 800: 202.185077ms (3972)
 800: 203.517794ms (3891)
 813: 202.252868ms (3930)
 1200: 208.142666ms (5097)
 1200: 223.430768ms (5428)
 1200: 220.116854ms (5439)
 1200: 222.68268ms (5326)
 1440: 242.595568ms (5203)
 1600: 305.24793ms (5148)
 1600: 347.119258ms (4147)
 1600: 365.609072ms (4869)
 1600: 373.862842ms (3990)
 1882: 362.321116ms (4307)
 2000: 381.745808ms (4874)
 2000: 411.721493ms (5463)
 2000: 407.099586ms (5141)
 2000: 426.357913ms (4570)
 2184: 418.562661ms (4892)
 2400: 457.59816ms (4528)
 2400: 454.36387ms (5363)
 2400: 466.733394ms (4876)
 2400: 463.128921ms (5164)
 2585: 457.770183ms (5249)
 2800: 500.760898ms (5861)
panic: EOF

erikdubbelboer · 2019-09-25T12:29:52Z

Something interesting, when I run the program using GOMAXPROCS=2 go run test3.go the latency is much lower for much longer:

...
 7045: 379.38203ms (13984)
 7200: 399.582396ms (21482)
 7200: 401.933279ms (14693)
 7200: 402.218023ms (21634)
 7200: 403.963008ms (14673)
 7600: 420.675264ms (22174)
 7600: 417.251859ms (15565)
 7600: 417.113607ms (17071)
 7600: 421.768307ms (21308)
 7851: 431.872806ms (15417)
 8000: 452.966165ms (16272)
 8000: 437.661982ms (21000)
 8000: 444.840521ms (19387)
 8000: 453.805204ms (16204)
 8400: 477.906078ms (16475)
 8400: 475.889074ms (17169)
 8400: 475.704451ms (17034)
 8400: 480.754576ms (17149)
 8464: 483.582626ms (17814)
panic: EOF

scanpatch · 2019-09-26T08:21:45Z

@erikdubbelboer, I also tested net/http and got results even worse than when working with fasthttp :)

Your examples and conclusions are very interesting. What should we do next? Try to reproduce similar conditions in other languages to prove that the problem is in golang?

erikdubbelboer · 2019-09-28T08:22:20Z

I want to try and reproduce this in C but I'll need some time to find out how do this with threads and epoll.

kirillDanshin · 2019-09-29T17:00:45Z

@erikdubbelboer libdill should handle this AFAIK

erikdubbelboer · 2019-10-11T09:45:16Z

I have wrote a similar program in C using libdill: https://gist.github.com/erikdubbelboer/f723725ec2c5a04aec31097be1aeca91

At 4800 (g|c)oroutines Go's latency is 334ms while libdill is 208ms, which is very close to the expected 200ms.

Seems like libdill doesn't suffer from a performance degradation with the same amount of (g|c)oroutines as Go. This seems to point in an issue with Go that should be possible to fix.

The Go program doesn't allocate anything either so it's not the garbage collector causing the extra overhead.

My next step will be to try and debug the performance of the Go program. Let me know if you guys see anything in my code that I'm missing that might be causing this.

vadyus · 2019-10-11T09:52:25Z

Take a look https://www.youtube.com/watch?v=h0s8CWpIKdg Might be same issue here

valyala · 2019-10-11T10:03:47Z

Go networking and timer subsystem performance don't scale due to global locks. I tried fixing timer subsystem in Go1.10, but it sometimes didn't work as expected as mentioned in the talk from the comment above. Scalability for timer subsystem should be fixed in Go1.14 - see the series of CLs starting from this one - https://go-review.googlesource.com/c/go/+/171821/16 .
Probably this will fix scalability for networking subsystem as well.

erikdubbelboer · 2019-10-19T16:36:36Z

I haven't tried 1.14 yet but upgrading to go1.13.3 has already lowered the latency for me, at 4800 goroutines from 334ms to 254ms. I'm using a Mac so I guess this commit is responsible: golang/go@81d995d

I'm now waiting for 1.14rc1 with all the commits @valyala mentioned merged.

vearutop · 2019-12-16T22:36:08Z

I've checked with my Ubuntu 19.10 laptop (i5 7200U 2 cores, 4 threads) and got somewhat similar results for go1.13.5 and latest gotip (go version devel +f7f9866 Mon Dec 16 20:38:31 2019 +0000 linux/amd64).

go1.13.5:

 27600: 447.098495ms (67356)
 27600: 448.895841ms (58116)
 27602: 431.380138ms (58622)
 27862: 436.042498ms (58507)
 28000: 435.410102ms (71065)
 28000: 448.287364ms (73307)
 28000: 471.628815ms (58352)
 28000: 452.509214ms (58940)
 28081: 451.909104ms (58595)
panic: dial tcp 127.0.0.1:8080: connect: cannot assign requested address

goroutine 112461 [running]:
main.main.func2.1(0xc000016300, 0xc0000162f0, 0xc0000162f8)
	/home/vearutop/dev/tmp/nettest.go:46 +0x1d9
created by main.main.func2
	/home/vearutop/dev/tmp/nettest.go:43 +0x5a
exit status 2

gotip:

 27600: 474.609679ms (55981)
 27602: 496.283602ms (44353)
 27945: 592.753063ms (41015)
 28000: 492.80866ms (60381)
 28000: 481.674884ms (64330)
 28000: 482.269362ms (63475)
 28094: 465.88271ms (66060)
panic: dial tcp 127.0.0.1:8080: connect: cannot assign requested address

goroutine 108121 [running]:
main.main.func2.1(0xc0000162d0, 0xc0000162c0, 0xc0000162c8)
	/home/vearutop/dev/tmp/nettest.go:46 +0x1d7
created by main.main.func2
	/home/vearutop/dev/tmp/nettest.go:43 +0x5a
exit status 2

erikdubbelboer · 2019-12-23T09:52:40Z

I did the same test. With GOMAXPROCS=1 they still perform about the same. So single core timer performance didn't change (and is still a lot worse than libdill).

But with GOMAXPROCS=16 (number of logical cores I have) the performance actually got a lot worse.
go1.13.5 at 9000 goroutines has an average latency of 202ms. While go1.14beta1 at 9000 goroutines has an average latency of 300ms (with this code).

I'm going to investigate this more when I have more time, but it seems like the networking subsystem or timers actually got a lot slower in 1.14.

@valyala

vearutop · 2019-12-23T11:13:42Z

Afaik there was a big change to scheduler with non-cooperative goroutine preemption (golang/go#24543) that can be causing performance regression under heavy concurrency.

erikdubbelboer · 2019-12-28T17:16:36Z

I have done some more testing on Linux as well and there the latency is always slightly lower with go1.14beta1 (And much much lower than my Mac). So it seems only Mac got slower, I created an issue here: golang/go#36298

As for the original issue. @r-romanov have you tried go1.14beta1 already? On an Ubuntu machine it performed really well for me.

erikdubbelboer · 2020-01-22T18:48:30Z

reuseport doesn't have anything to do with performance. It just allows you to dial or listen multiple times from the same address (allowing duplicate address tuples).

scanpatch · 2020-06-04T16:37:07Z

Now i am using golang 1.14.3 and the situation is the same.

I plan to rewrite my tools in C or Rust and abandon golang for crawling sites. Very sad :(

avierr · 2020-08-12T12:19:31Z

Now i am using golang 1.14.3 and the situation is the same.

I plan to rewrite my tools in C or Rust and abandon golang for crawling sites. Very sad :(

golang/go#36365 as you can see, it will be a while before the pre-emptive scheduler becomes usable and as we can't manage threads on our own, so you can't really use Go at that scale, but here is one library that you could use:
https://github.com/liamzdenek/go-pthreads

Elixir/Erlang has pre-emptive scheduling so might suit your task instead of manual thread management in C

erikdubbelboer added the pending/submitter-response label Sep 20, 2019

erikdubbelboer added the pending/investigation label Sep 22, 2019

erikdubbelboer removed the pending/submitter-response label Sep 24, 2019

dumblob mentioned this issue Feb 29, 2020

V Concurrency Considerations vlang/v#3814

Closed

erikdubbelboer closed this as completed Sep 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Too many timeouts when working with high concurrency #656

Too many timeouts when working with high concurrency #656

scanpatch commented Sep 18, 2019

erikdubbelboer commented Sep 20, 2019

scanpatch commented Sep 20, 2019

kirillDanshin commented Sep 21, 2019

iapyeh commented Sep 21, 2019

kirillDanshin commented Sep 21, 2019

erikdubbelboer commented Sep 21, 2019

scanpatch commented Sep 22, 2019 •

edited

Loading

erikdubbelboer commented Sep 22, 2019

scanpatch commented Sep 22, 2019

erikdubbelboer commented Sep 24, 2019

erikdubbelboer commented Sep 25, 2019

erikdubbelboer commented Sep 25, 2019

scanpatch commented Sep 26, 2019

erikdubbelboer commented Sep 28, 2019

kirillDanshin commented Sep 29, 2019

erikdubbelboer commented Oct 11, 2019

vadyus commented Oct 11, 2019

valyala commented Oct 11, 2019

erikdubbelboer commented Oct 19, 2019

vearutop commented Dec 16, 2019

erikdubbelboer commented Dec 23, 2019

vearutop commented Dec 23, 2019

erikdubbelboer commented Dec 28, 2019

erikdubbelboer commented Jan 22, 2020

scanpatch commented Jun 4, 2020

avierr commented Aug 12, 2020 •

edited

Loading

Too many timeouts when working with high concurrency #656

Too many timeouts when working with high concurrency #656

Comments

scanpatch commented Sep 18, 2019

erikdubbelboer commented Sep 20, 2019

scanpatch commented Sep 20, 2019

kirillDanshin commented Sep 21, 2019

iapyeh commented Sep 21, 2019

kirillDanshin commented Sep 21, 2019

erikdubbelboer commented Sep 21, 2019

scanpatch commented Sep 22, 2019 • edited Loading

erikdubbelboer commented Sep 22, 2019

scanpatch commented Sep 22, 2019

erikdubbelboer commented Sep 24, 2019

erikdubbelboer commented Sep 25, 2019

erikdubbelboer commented Sep 25, 2019

scanpatch commented Sep 26, 2019

erikdubbelboer commented Sep 28, 2019

kirillDanshin commented Sep 29, 2019

erikdubbelboer commented Oct 11, 2019

vadyus commented Oct 11, 2019

valyala commented Oct 11, 2019

erikdubbelboer commented Oct 19, 2019

vearutop commented Dec 16, 2019

erikdubbelboer commented Dec 23, 2019

vearutop commented Dec 23, 2019

erikdubbelboer commented Dec 28, 2019

erikdubbelboer commented Jan 22, 2020

scanpatch commented Jun 4, 2020

avierr commented Aug 12, 2020 • edited Loading

scanpatch commented Sep 22, 2019 •

edited

Loading

avierr commented Aug 12, 2020 •

edited

Loading