Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use DPMCPs for communicating with MC / fix VFIO guest #2

Closed
mcbridematt opened this issue Apr 26, 2022 · 2 comments
Closed

Use DPMCPs for communicating with MC / fix VFIO guest #2

mcbridematt opened this issue Apr 26, 2022 · 2 comments
Assignees
@dsalychev
Labels
bug Something isn't working enhancement New feature or request

Comments

@mcbridematt
Copy link

Under VFIO passthrough for DPAA2, most MC commands have to be run through DPMCP's instead of the DPRC root container.

This requirement is enforced by QEMU which has a security filter:
https://source.codeaurora.org/external/qoriq/qoriq-components/qemu/tree/hw/vfio/fsl_mc.c?h=integration&id=14fda5a42df6c72e890d6a97ff88c5852172604b#n688

If you attempt to do DPBP, DPIO or most other object instructions through the DPRC, you get a 0x3 (authentication error) response generated by QEMU:

dpaa2_mc0: mem 0x4040000000-0x404000ffff on ofwbus0
dpaa2_rc0: on dpaa2_mc0
dpaa2_rc0: MC firmware version: 10.20.4
dpaa2_bp0: at dpbp (id=0) on dpaa2_rc0
dpaa2_bp0: Failed to reset DPBP: id=0, error=3
device_attach: dpaa2_bp0 attach returned 6
dpaa2_io0: <DPAA2 I/O> iomem 0x4048000000-0x404800ffff,0x4044000000-0x404400ffff at dpio (id=0) on dpaa2_rc0
dpaa2_io0: Failed to reset DPIO: id=0, error=3
device_attach: dpaa2_io0 attach returned 6
dpaa2_con0: at dpcon (id=2) on dpaa2_rc0
dpaa2_con0: Failed to reset DPCON: id=2, error=3
device_attach: dpaa2_con0 attach returned 6
dpaa2_con1: at dpcon (id=0) on dpaa2_rc0
dpaa2_con1: Failed to reset DPCON: id=0, error=3
@dsalychev dsalychev self-assigned this Apr 26, 2022
@dsalychev dsalychev added bug Something isn't working enhancement New feature or request labels Apr 26, 2022
dsalychev added a commit that referenced this issue Apr 29, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
855b2c4
dsalychev added a commit that referenced this issue Apr 29, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
66ed301
@dsalychev
Copy link

@mcbridematt Please, try 2cab65b or any commit after that one. I've implemented a mechanism to discover and allocate DPMCPs.

@mcbridematt
Copy link
Author

mcbridematt commented May 17, 2022 •
edited
Loading

Thanks very much for that!
Under VFIO it can finish boot now.

dpaa2_rc0: <DPAA2 Resource Container> on dpaa2_mc0
dpaa2_rc0: MC firmware version: 10.20.4
dpaa2_rc0: Resource container ID: 2
dpaa2_rc0: Objects in container: 8
dpaa2_rc0: Isolation context ID: 0
dpaa2_mcp0: <DPAA2 MC portal> iomem 0x4040020000-0x404002ffff at dpmcp (id=28) on dpaa2_rc0
dpaa2_mcp1: <DPAA2 MC portal> iomem 0x4040010000-0x404001ffff at dpmcp (id=27) on dpaa2_rc0
dpaa2_rc0: dpaa2_rc_discover: skip unsupported DPAA2 object: idx=2
dpaa2_bp0: <DPAA2 Buffer Pool> dpmcp (id=27) at dpbp (id=0) on dpaa2_rc0
dpaa2_io0: <DPAA2 I/O> iomem 0x4048000000-0x404800ffff,0x4044000000-0x404400ffff dpmcp (id=28) at dpio (id=0) on dpaa2_rc0
dpaa2_io0: using IRQ 40 for MSI
dpaa2_io0: dpio_id=0, swp_id=9, chan_mode=local_channel, notif_priors=2, swp_version=0x4010001
dpaa2_con0: <DPAA2 Concentrator> dpmcp (id=28) at dpcon (id=2) on dpaa2_rc0
dpaa2_con0: chan_id=3, priorities=2
dpaa2_con1: <DPAA2 Concentrator> dpmcp (id=27) at dpcon (id=0) on dpaa2_rc0
dpaa2_con1: chan_id=2, priorities=2
dpaa2_rc0: dpaa2_rc_discover: skip unsupported DPAA2 object: idx=2
dpaa2_ni0: <DPAA2 Network Interface> dpio (id=0) dpbp (id=0) dpcon (id=0) dpmcp (id=28) at dpni (id=1) on dpaa2_rc0
dpaa2_ni0: options=0x0 queues=1 tx_channels=0 wriop_version=0x422
dpaa2_ni0:      traffic classes: rx=1 tx=1 cgs_groups=1
dpaa2_ni0:      table entries: mac=16 vlan=0 qos=0 fs=64
dpaa2_ni0:      key sizes: qos=0 fs=56
dpaa2_ni0: Rx/Tx buffers: size=9216, alignment=64
dpaa2_ni0: Tx data offset=192
dpaa2_ni0: connected to dpmac (id=3)
dpaa2_ni0: dpaa2_ni_setup: failed to open connected DPMAC: 3 (assuming in other DPRC)
dpaa2_ni0: connected DPMAC is in FIXED mode
dpaa2_ni0: Ingress traffic classification is not supported
dpaa2_ni0: channels=1
dpaa2_ni0: channel: dpio_id=0 dpcon_id=0 chan_id=2, priorities=2
dpaa2_ni0: Ingress traffic distribution not supported
dpaa2_ni0: using IRQ 41 for MSI

dpni0 isn't receiving any frames (I can see frames coming out), but I will troubleshoot this another time.

It also appears we need to treat an unreachable (in other DPRC) DPNI partner as a 'fixed' link. I'll send a patch once I figure out why the ingress isn't working.

(edit: maybe due to an incorrect ingress filter?

dev.dpaa2_ni.0.stats.in_filtered_frames: 328
dev.dpaa2_ni.0.stats.in_discarded_frames: 0
dev.dpaa2_ni.0.stats.in_nobuf_discards: 145

I'll look at it again tomorrow
)

dsalychev added a commit that referenced this issue Jun 3, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
970d30d
dsalychev added a commit that referenced this issue Jun 3, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
67ffce9
dsalychev pushed a commit that referenced this issue Aug 19, 2022
@AlexanderChernikov
routing: refactor #2
dedeec1 
* Use same filter func (rib_filter_f_t) for nexhtop groups to
 simplify callbacks.
* simplify conditional route deletion & remove the need to pass
 rt_addrinfo to the low-level deletion functions
* speedup rib_walk_del() by removing an additional per-prefix lookup

Differential Revision: https://reviews.freebsd.org/D36071
MFC after:	1 month
dsalychev added a commit that referenced this issue Aug 19, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
4660624
dsalychev added a commit that referenced this issue Aug 19, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
44a1f12
dsalychev added a commit that referenced this issue Aug 20, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
531a648
dsalychev added a commit that referenced this issue Aug 20, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
254267f
dsalychev added a commit that referenced this issue Aug 29, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
1d64481
dsalychev added a commit that referenced this issue Aug 29, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
5365c15
dsalychev added a commit that referenced this issue Sep 9, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
92bdbd8
dsalychev added a commit that referenced this issue Sep 9, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
2870689
dsalychev added a commit that referenced this issue Sep 20, 2022
@dsalychev
dpaa2: Debug output to trace PIC lookup #2.
cf08a8c
dsalychev added a commit that referenced this issue Sep 20, 2022
@dsalychev
dpaa2: Tweaks to prevent channels re-arming failure #2.
6430c92
dsalychev pushed a commit that referenced this issue Apr 3, 2023
@rob-wing
FreeBSD: don't verify recycled vnode for zfs control directory
28251d8 
Under certain loads, the following panic is hit:

    panic: page fault
    KDB: stack backtrace:
    #0 0xffffffff805db025 at kdb_backtrace+0x65
    #1 0xffffffff8058e86f at vpanic+0x17f
    #2 0xffffffff8058e6e3 at panic+0x43
    #3 0xffffffff808adc15 at trap_fatal+0x385
    #4 0xffffffff808adc6f at trap_pfault+0x4f
    #5 0xffffffff80886da8 at calltrap+0x8
    #6 0xffffffff80669186 at vgonel+0x186
    #7 0xffffffff80669841 at vgone+0x31
    #8 0xffffffff8065806d at vfs_hash_insert+0x26d
    #9 0xffffffff81a39069 at sfs_vgetx+0x149
    #10 0xffffffff81a39c54 at zfsctl_snapdir_lookup+0x1e4
    #11 0xffffffff8065a28c at lookup+0x45c
    #12 0xffffffff806594b9 at namei+0x259
    #13 0xffffffff80676a33 at kern_statat+0xf3
    #14 0xffffffff8067712f at sys_fstatat+0x2f
    #15 0xffffffff808ae50c at amd64_syscall+0x10c
    #16 0xffffffff808876bb at fast_syscall_common+0xf8

The page fault occurs because vgonel() will call VOP_CLOSE() for active
vnodes. For this reason, define vop_close for zfsctl_ops_snapshot. While
here, define vop_open for consistency.

After adding the necessary vop, the bug progresses to the following
panic:

    panic: VERIFY3(vrecycle(vp) == 1) failed (0 == 1)
    cpuid = 17
    KDB: stack backtrace:
    #0 0xffffffff805e29c5 at kdb_backtrace+0x65
    #1 0xffffffff8059620f at vpanic+0x17f
    #2 0xffffffff81a27f4a at spl_panic+0x3a
    #3 0xffffffff81a3a4d0 at zfsctl_snapshot_inactive+0x40
    #4 0xffffffff8066fdee at vinactivef+0xde
    #5 0xffffffff80670b8a at vgonel+0x1ea
    #6 0xffffffff806711e1 at vgone+0x31
    #7 0xffffffff8065fa0d at vfs_hash_insert+0x26d
    #8 0xffffffff81a39069 at sfs_vgetx+0x149
    #9 0xffffffff81a39c54 at zfsctl_snapdir_lookup+0x1e4
    #10 0xffffffff80661c2c at lookup+0x45c
    #11 0xffffffff80660e59 at namei+0x259
    #12 0xffffffff8067e3d3 at kern_statat+0xf3
    #13 0xffffffff8067eacf at sys_fstatat+0x2f
    #14 0xffffffff808b5ecc at amd64_syscall+0x10c
    #15 0xffffffff8088f07b at fast_syscall_common+0xf8

This is caused by a race condition that can occur when allocating a new
vnode and adding that vnode to the vfs hash. If the newly created vnode
loses the race when being inserted into the vfs hash, it will not be
recycled as its usecount is greater than zero, hitting the above
assertion.

Fix this by dropping the assertion.

FreeBSD-issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252700
Reviewed-by: Andriy Gapon <avg@FreeBSD.org>
Reviewed-by: Mateusz Guzik <mjguzik@gmail.com>
Reviewed-by: Alek Pinchuk <apinchuk@axcient.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Rob Wing <rob.wing@klarasystems.com>
Co-authored-by: Rob Wing <rob.wing@klarasystems.com>
Submitted-by: Klara, Inc.
Sponsored-by: rsync.net
Closes #14501
dsalychev pushed a commit that referenced this issue Jun 9, 2023
@AlexanderChernikov
ifconfig: fix warnings #2
0c2beef 
Fix all -Wparameter-unused and cast alignment

Differential Revision: https://reviews.freebsd.org/D40303
MFC after:	2 weeks
dsalychev pushed a commit that referenced this issue Aug 3, 2023
@kprovost
mroute: avoid calling if_allmulti with the lock held
680ad06 
Avoid locking issues when if_allmulti() calls the driver's if_ioctl,
because that may acquire sleepable locks (while we hold a non-sleepable
rwlock).

Fortunately there's no pressing need to hold the mroute lock while we
do this, so we can postpone the call slightly, until after we've
released the lock.

This avoids the following WITNESS warning (with iflib drivers):

	lock order reversal: (sleepable after non-sleepable)
	 1st 0xffffffff82f64960 IPv4 multicast forwarding (IPv4 multicast forwarding, rw) @ /usr/src/sys/netinet/ip_mroute.c:1050
	 2nd 0xfffff8000480f180 iflib ctx lock (iflib ctx lock, sx) @ /usr/src/sys/net/iflib.c:4525
	lock order IPv4 multicast forwarding -> iflib ctx lock attempted at:
	#0 0xffffffff80bbd6ce at witness_checkorder+0xbbe
	#1 0xffffffff80b56d10 at _sx_xlock+0x60
	#2 0xffffffff80c9ce5c at iflib_if_ioctl+0x2dc
	#3 0xffffffff80c7c395 at if_setflag+0xe5
	#4 0xffffffff82f60a0e at del_vif_locked+0x9e
	#5 0xffffffff82f5f0d5 at X_ip_mrouter_set+0x265
	#6 0xffffffff80bfd402 at sosetopt+0xc2
	#7 0xffffffff80c02105 at kern_setsockopt+0xa5
	#8 0xffffffff80c02054 at sys_setsockopt+0x24
	#9 0xffffffff81046be8 at amd64_syscall+0x138
	#10 0xffffffff8101930b at fast_syscall_common+0xf8

See also:	https://redmine.pfsense.org/issues/12079
Reviewed by:	mjg
Sponsored by:	Rubicon Communications, LLC ("Netgate")
Differential Revision:	https://reviews.freebsd.org/D41209
dsalychev pushed a commit that referenced this issue Feb 1, 2024
@kevans91
kern: console: make /dev/console backing console more predictable
8ec6d7b 
Specifically, altering the console list with conscontrol has some weird
behavior:

1. If you remove the first configured console, /dev/console will become
  unconfigured
2. Any console added becomes the /dev/console

In a multicons situation, #1 is clearly a bug and #2 is perhaps slightly
less clear.  If we have ttyu0, ttyv0, then it seems obvious that one
would want ttyv0 to take over the console if ttyu0 is removed.  If we
add ttyu0 back in, then it's debatable whether it should take over the
console or not.

Fix it now to make the /dev/console selection more FIFO-ish, with
respect to how conscontrol affects it.  A `primary` verb for
conscontrol(8) might be a good addition.
dsalychev pushed a commit that referenced this issue Nov 18, 2024
@jlduran
ipfilter: Avoid stopping with a lock held
1fa6daa 
Avoid calling _callout_stop_safe with a non-sleepable lock held when
detaching by initializing callout_init_rw() with CALLOUT_SHAREDLOCK.

It avoids the following WITNESS warning when stopping the service:

    # service ipfilter stop
    calling _callout_stop_safe with the following non-sleepable locks held:
    shared rw ipf filter load/unload mutex (ipf filter load/unload mutex) r = 0 (0xffff0000417c7530) locked @ /usr/src/sys/netpfil/ipfilter/netinet/fil.c:7926
    stack backtrace:
    #0 0xffff00000052d394 at witness_debugger+0x60
    #1 0xffff00000052e620 at witness_warn+0x404
    #2 0xffff0000004d4ffc at _callout_stop_safe+0x8c
    #3 0xffff0000f7236674 at ipfdetach+0x3c
    #4 0xffff0000f723fa4c at ipf_ipf_ioctl+0x788
    #5 0xffff0000f72367e0 at ipfioctl+0x144
    #6 0xffff00000034abd8 at devfs_ioctl+0x100
    #7 0xffff0000005c66a0 at vn_ioctl+0xbc
    #8 0xffff00000034b2cc at devfs_ioctl_f+0x24
    #9 0xffff0000005331ec at kern_ioctl+0x2e0
    #10 0xffff000000532eb4 at sys_ioctl+0x140
    #11 0xffff000000880480 at do_el0_sync+0x604
    #12 0xffff0000008579ac at handle_el0_sync+0x4c

PR:		282478
Suggested by:	markj
Reviewed by:	cy
Approved by:	emaste (mentor)
MFC after:	1 week
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dsalychev dsalychev

Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dsalychev @mcbridematt