frr: T5239: fix process startup order #2245

c-po · 2023-09-11T20:02:23Z

Change Summary

Reuse existing utility functions to check if a boot is ongoing (boot_configuration_complete())
Run system_frr.py script to configure FRR daemon before initial launch
Add safety net to always have FRR running on the system

This does yet not solve the error in T5239 but it's a small step towards the solution.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Code style update (formatting, renaming)
Refactoring (no functional changes)
Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
Other (please describe):

Related Task(s)

https://vyos.dev/T5239

Related PRs

frr: T5239: remove daemons.conf generated by chroot hook vyos-build#401

Component(s) name

FRR and vyos-router startup

Proposed changes

How to test

Smoketests

Checklist:

I have read the CONTRIBUTING document
I have linked this PR to one or more Phabricator Task(s)
I have run the components SMOKETESTS if applicable
My commit headlines contain a valid Task id
My change requires a change to the documentation
I have updated the documentation accordingly

- Reuse existing utility functions to check if a boot is ongoing (boot_configuration_complete()) - Run system_frr.py script to configure FRR daemon before initial launch - Add safety net to always have FRR running on the system This does yet not solve the error in T5239 but it's a small step towards the solution.

Daemon configuration is now generated during boot via vyos-1x repo. See vyos/vyos-1x#2245

c-po · 2023-09-13T11:42:01Z

@Mergifyio backport sagitta

mergify · 2023-09-13T11:42:05Z

backport sagitta

✅ Backports have been created

#2254 frr: T5239: fix process startup order (backport #2245) has been created for branch sagitta

Daemon configuration is now generated during boot via vyos-1x repo. See vyos/vyos-1x#2245 (cherry picked from commit a9a1ca3)

Apachez- · 2023-09-14T06:54:39Z

Looks like smoketests regarding rpki went unhappy:

Could https://vyos.dev/T2044 be related to the failed nightly build from last night?

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6179287424/job/16773987622#step:10:28483

DEBUG - test_route_map (__main__.TestPolicy.test_route_map) ... FAIL

https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6179287424/job/16773987622#step:10:28642

DEBUG - test_rpki (__main__.TestProtocolsRPKI.test_rpki) ... ERROR

c-po · 2023-09-14T06:56:39Z

RPKI has a general bug T2044 so I'd ignore this one here and focus on resolving T2044 which should fix this, too

Apachez- · 2023-09-14T08:16:58Z

Attempt is made by PR: #2264

frr: T5239: fix process startup order (backport #2245)

Apachez- · 2023-09-15T02:28:51Z

Smoketests seems unhappy even during boot regarding frr:

2023-09-15T01:32:41.4901436Z DEBUG - [   17.759778] vyos-router[1188]: Waiting for NICs to settle down: settled in 0sec..
2023-09-15T01:32:41.4901887Z DEBUG - [   20.407080] vyos-router[1547]: VyOS had an issue completing a command.
2023-09-15T01:32:41.4902438Z DEBUG - [   20.408063] vyos-router[1547]: We are sorry that you encountered a problem while using VyOS.
2023-09-15T01:32:41.4902974Z DEBUG - [   20.409261] vyos-router[1547]: There are a few things you can do to help us (and yourself):
2023-09-15T01:32:41.4903541Z DEBUG - [   20.410435] vyos-router[1547]: - Contact us using the online help desk if you have a subscription:
2023-09-15T01:32:41.4903939Z DEBUG - [   20.411662] vyos-router[1547]:   https://support.vyos.io/
2023-09-15T01:32:41.4904505Z DEBUG - [   20.412447] vyos-router[1547]: - Make sure you are running the latest version of VyOS available at:
2023-09-15T01:32:41.4904909Z DEBUG - [   20.413700] vyos-router[1547]:   https://vyos.net/get/
2023-09-15T01:32:41.4905451Z DEBUG - [   20.414453] vyos-router[1547]: - Consult the community forum to see how to handle this issue:
2023-09-15T01:32:41.4905826Z DEBUG - [   20.415607] vyos-router[1547]:   https://forum.vyos.io
2023-09-15T01:32:41.4906364Z DEBUG - [   20.416374] vyos-router[1547]: - Join us on Slack where our users exchange help and advice:
2023-09-15T01:32:41.4906754Z DEBUG - [   20.417540] vyos-router[1547]:   https://vyos.slack.com
2023-09-15T01:32:41.4907338Z DEBUG - [   20.418295] vyos-router[1547]: When reporting problems, please include as much information as possible:
2023-09-15T01:32:41.4907903Z DEBUG - [   20.419543] vyos-router[1547]: - do not obfuscate any data (feel free to contact us privately if your
2023-09-15T01:32:41.4908314Z DEBUG - [   20.420768] vyos-router[1547]:   business policy requires it)
2023-09-15T01:32:41.4908811Z DEBUG - [   20.421572] vyos-router[1547]: - and include all the information presented below
2023-09-15T01:32:41.4909253Z DEBUG - [   20.422602] vyos-router[1547]: Report time:      2023-09-15 00:36:52
2023-09-15T01:32:41.4909713Z DEBUG - [   20.423472] vyos-router[1547]: Image version:    VyOS 1.5-rolling-202309150021
2023-09-15T01:32:41.4910092Z DEBUG - [   20.424477] vyos-router[1547]: Release train:    current
2023-09-15T01:32:41.4910519Z DEBUG - [   20.425207] vyos-router[1547]: Built by:         autobuild@vyos.net
2023-09-15T01:32:41.4910973Z DEBUG - [   20.426058] vyos-router[1547]: Built on:         Fri 15 Sep 2023 00:23 UTC
2023-09-15T01:32:41.4911473Z DEBUG - [   20.427015] vyos-router[1547]: Build UUID:       f931b828-4e72-44e0-a1cf-a25efeb44cfe
2023-09-15T01:32:41.4911897Z DEBUG - [   20.428058] vyos-router[1547]: Build commit ID:  e7c29ed7be44e3
2023-09-15T01:32:41.4912278Z DEBUG - [   20.428868] vyos-router[1547]: Architecture:     x86_64
2023-09-15T01:32:41.4912695Z DEBUG - [   20.429599] vyos-router[1547]: Boot via:         installed image
2023-09-15T01:32:41.4913177Z DEBUG - [   20.430446] vyos-router[1547]: System type:      KVM guest
2023-09-15T01:32:41.4916780Z DEBUG - [   20.431241] vyos-router[1547]: Hardware vendor:  QEMU
2023-09-15T01:32:41.4917446Z DEBUG - [   20.431942] vyos-router[1547]: Hardware model:   Standard PC (i440FX + PIIX, 1996)
2023-09-15T01:32:41.4917812Z DEBUG - [   20.432980] vyos-router[1547]: Hardware S/N:
2023-09-15T01:32:41.4918319Z DEBUG - [   20.433607] vyos-router[1547]: Hardware UUID:    d27cf29e-4419-4407-8f82-dc73d1acd184
2023-09-15T01:32:41.4918754Z DEBUG - [   20.434668] vyos-router[1547]: Traceback (most recent call last):
2023-09-15T01:32:41.4919325Z DEBUG - [   20.435521] vyos-router[1547]:   File "/usr/libexec/vyos/conf_mode/system-ip.py", line 134, in <module>
2023-09-15T01:32:41.4919677Z DEBUG - [   20.436801] vyos-router[1547]:     apply(c)
2023-09-15T01:32:41.4920221Z DEBUG - [   20.437429] vyos-router[1547]:   File "/usr/libexec/vyos/conf_mode/system-ip.py", line 123, in apply
2023-09-15T01:32:41.4920890Z DEBUG - [   20.438704] vyos-router[1547]:     frr_cfg.load_configuration(zebra_daemon)
2023-09-15T01:32:41.4921504Z DEBUG - [   20.439694] vyos-router[1547]:   File "/usr/lib/python3/dist-packages/vyos/frr.py", line 438, in load_configuration
2023-09-15T01:32:41.4922037Z DEBUG - [   20.441126] vyos-router[1547]:     self.imported_config = get_configuration(daemon=daemon)
2023-09-15T01:32:41.4922506Z DEBUG - [   20.442251] vyos-router[1547]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-15T01:32:41.4923108Z DEBUG - [   20.443371] vyos-router[1547]:   File "/usr/lib/python3/dist-packages/vyos/frr.py", line 152, in get_configuration
2023-09-15T01:32:41.4923525Z DEBUG - [   20.444742] vyos-router[1547]:     raise OSError(code, output)
2023-09-15T01:32:41.4924102Z DEBUG - [   20.445582] vyos-router[1547]: PermissionError: [Errno 1] Exiting: failed to connect to any daemons.
2023-09-15T01:32:41.4924557Z DEBUG - [   20.446937] vyos-router[1188]: could not reset system IPv4 options ... failed!
2023-09-15T01:32:41.4925021Z DEBUG - [   20.583998] vyos-router[1570]: VyOS had an issue completing a command.
2023-09-15T01:32:41.4925567Z DEBUG - [   20.585099] vyos-router[1570]: We are sorry that you encountered a problem while using VyOS.
2023-09-15T01:32:41.4926102Z DEBUG - [   20.586320] vyos-router[1570]: There are a few things you can do to help us (and yourself):
2023-09-15T01:32:41.4926662Z DEBUG - [   20.587543] vyos-router[1570]: - Contact us using the online help desk if you have a subscription:
2023-09-15T01:32:41.4927075Z DEBUG - [   20.588814] vyos-router[1570]:   https://support.vyos.io/
2023-09-15T01:32:41.4942333Z DEBUG - [   20.589635] vyos-router[1570]: - Make sure you are running the latest version of VyOS available at:
2023-09-15T01:32:41.4942868Z DEBUG - [   20.590901] vyos-router[1570]:   https://vyos.net/get/
2023-09-15T01:32:41.4943440Z DEBUG - [   20.591652] vyos-router[1570]: - Consult the community forum to see how to handle this issue:
2023-09-15T01:32:41.4943835Z DEBUG - [   20.592782] vyos-router[1570]:   https://forum.vyos.io
2023-09-15T01:32:41.4944380Z DEBUG - [   20.593525] vyos-router[1570]: - Join us on Slack where our users exchange help and advice:
2023-09-15T01:32:41.4944775Z DEBUG - [   20.594654] vyos-router[1570]:   https://vyos.slack.com
2023-09-15T01:32:41.4945371Z DEBUG - [   20.595435] vyos-router[1570]: When reporting problems, please include as much information as possible:
2023-09-15T01:32:41.4945931Z DEBUG - [   20.596691] vyos-router[1570]: - do not obfuscate any data (feel free to contact us privately if your
2023-09-15T01:32:41.4946488Z DEBUG - [   20.597935] vyos-router[1570]:   business policy requires it)
2023-09-15T01:32:41.4946979Z DEBUG - [   20.598774] vyos-router[1570]: - and include all the information presented below
2023-09-15T01:32:41.4947426Z DEBUG - [   20.599812] vyos-router[1570]: Report time:      2023-09-15 00:36:52
2023-09-15T01:32:41.4947887Z DEBUG - [   20.600711] vyos-router[1570]: Image version:    VyOS 1.5-rolling-202309150021
2023-09-15T01:32:41.4948537Z DEBUG - [   20.601728] vyos-router[1570]: Release train:    current
2023-09-15T01:32:41.4948971Z DEBUG - [   20.602489] vyos-router[1570]: Built by:         autobuild@vyos.net
2023-09-15T01:32:41.4949430Z DEBUG - [   20.603386] vyos-router[1570]: Built on:         Fri 15 Sep 2023 00:23 UTC
2023-09-15T01:32:41.4949928Z DEBUG - [   20.604358] vyos-router[1570]: Build UUID:       f931b828-4e72-44e0-a1cf-a25efeb44cfe
2023-09-15T01:32:41.4950359Z DEBUG - [   20.605423] vyos-router[1570]: Build commit ID:  e7c29ed7be44e3
2023-09-15T01:32:41.4950746Z DEBUG - [   20.606303] vyos-router[1570]: Architecture:     x86_64
2023-09-15T01:32:41.4951163Z DEBUG - [   20.607129] vyos-router[1570]: Boot via:         installed image
2023-09-15T01:32:41.4951534Z DEBUG - [   20.607970] vyos-router[1570]: System type:      KVM guest
2023-09-15T01:32:41.4951919Z DEBUG - [   20.608740] vyos-router[1570]: Hardware vendor:  QEMU
2023-09-15T01:32:41.4952533Z DEBUG - [   20.609459] vyos-router[1570]: Hardware model:   Standard PC (i440FX + PIIX, 1996)
2023-09-15T01:32:41.4952895Z DEBUG - [   20.610493] vyos-router[1570]: Hardware S/N:
2023-09-15T01:32:41.4953400Z DEBUG - [   20.611122] vyos-router[1570]: Hardware UUID:    d27cf29e-4419-4407-8f82-dc73d1acd184
2023-09-15T01:32:41.4954596Z DEBUG - [   20.612195] vyos-router[1570]: Traceback (most recent call last):
2023-09-15T01:32:41.4968360Z DEBUG - [   20.613043] vyos-router[1570]:   File "/usr/libexec/vyos/conf_mode/system-ipv6.py", line 112, in <module>
2023-09-15T01:32:41.4968856Z DEBUG - [   20.614306] vyos-router[1570]:     apply(c)
2023-09-15T01:32:41.4969436Z DEBUG - [   20.614888] vyos-router[1570]:   File "/usr/libexec/vyos/conf_mode/system-ipv6.py", line 101, in apply
2023-09-15T01:32:41.4969968Z DEBUG - [   20.616144] vyos-router[1570]:     frr_cfg.load_configuration(zebra_daemon)
2023-09-15T01:32:41.4970590Z DEBUG - [   20.617118] vyos-router[1570]:   File "/usr/lib/python3/dist-packages/vyos/frr.py", line 438, in load_configuration
2023-09-15T01:32:41.4971144Z DEBUG - [   20.618516] vyos-router[1570]:     self.imported_config = get_configuration(daemon=daemon)
2023-09-15T01:32:41.4971727Z DEBUG - [   20.619617] vyos-router[1570]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-09-15T01:32:41.4972344Z DEBUG - [   20.620707] vyos-router[1570]:   File "/usr/lib/python3/dist-packages/vyos/frr.py", line 152, in get_configuration
2023-09-15T01:32:41.4972767Z DEBUG - [   20.622115] vyos-router[1570]:     raise OSError(code, output)
2023-09-15T01:32:41.4973348Z DEBUG - [   20.622933] vyos-router[1570]: PermissionError: [Errno 1] Exiting: failed to connect to any daemons.
2023-09-15T01:32:41.4973811Z DEBUG - [   20.624202] vyos-router[1188]: could not reset system IPv6 options ... failed!
2023-09-15T01:32:41.4974225Z DEBUG - [   21.311825] vyos-router[1188]: Mounting VyOS Config...done.
2023-09-15T01:32:41.4974682Z DEBUG - [   25.946534] vyos-router[1188]: Starting VyOS router: migrate configure.
2023-09-15T01:32:41.4975072Z DEBUG - [   26.632550] vyos-config[1199]: Configuration success
2023-09-15T01:32:41.4975287Z DEBUG - 
2023-09-15T01:32:41.4975594Z DEBUG - Welcome to VyOS - vyos ttyS0

c-po · 2023-09-15T05:29:19Z

Error was reported by Jenkins and is fixed in 22d5cd4

github-actions bot added the current label Sep 11, 2023

github-actions bot assigned c-po Sep 11, 2023

vyosbot requested review from a team, dmbaturin, sarthurdev, zdc, jestabro and sever-sever and removed request for a team September 11, 2023 20:02

c-po added a commit to c-po/vyos-build that referenced this pull request Sep 11, 2023

frr: T5239: remove daemons.conf generated by chroot hook

a9a1ca3

Daemon configuration is now generated during boot via vyos-1x repo. See vyos/vyos-1x#2245

c-po mentioned this pull request Sep 11, 2023

frr: T5239: remove daemons.conf generated by chroot hook vyos/vyos-build#401

Merged

11 tasks

frr: T5239: T2061: prevent writing logs to /var/log/frr/frr.log

d4b9b2a

sever-sever approved these changes Sep 12, 2023

View reviewed changes

zdc approved these changes Sep 13, 2023

View reviewed changes

c-po merged commit 1bc8087 into vyos:current Sep 13, 2023
7 checks passed

mergify bot mentioned this pull request Sep 13, 2023

frr: T5239: fix process startup order (backport #2245) #2254

Merged

mergify bot pushed a commit to vyos/vyos-build that referenced this pull request Sep 13, 2023

frr: T5239: remove daemons.conf generated by chroot hook

5f3a5ea

Daemon configuration is now generated during boot via vyos-1x repo. See vyos/vyos-1x#2245 (cherry picked from commit a9a1ca3)

c-po added a commit that referenced this pull request Sep 14, 2023

Merge pull request #2254 from vyos/mergify/bp/sagitta/pr-2245

c5498ec

frr: T5239: fix process startup order (backport #2245)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

frr: T5239: fix process startup order #2245

frr: T5239: fix process startup order #2245

c-po commented Sep 11, 2023 •

edited

c-po commented Sep 13, 2023

mergify bot commented Sep 13, 2023 •

edited

Apachez- commented Sep 14, 2023

c-po commented Sep 14, 2023

Apachez- commented Sep 14, 2023

Apachez- commented Sep 15, 2023

c-po commented Sep 15, 2023

frr: T5239: fix process startup order #2245

frr: T5239: fix process startup order #2245

Conversation

c-po commented Sep 11, 2023 • edited

Change Summary

Types of changes

Related Task(s)

Related PRs

Component(s) name

Proposed changes

How to test

Checklist:

c-po commented Sep 13, 2023

mergify bot commented Sep 13, 2023 • edited

✅ Backports have been created

Apachez- commented Sep 14, 2023

c-po commented Sep 14, 2023

Apachez- commented Sep 14, 2023

Apachez- commented Sep 15, 2023

c-po commented Sep 15, 2023

c-po commented Sep 11, 2023 •

edited

mergify bot commented Sep 13, 2023 •

edited