axgbe: Various link stability and module compatibilty improvements

[swhite2]

Move the phy_stop() routine to if_detach() to prevent link interruptions when configuring the interface. Accompanying this is a sanity check using phy_started, which was already there but remained unused. We do not move phy_start(), as the logic there is needed for any init routine, be it attach or start.

Also bring in the linux PMA_PLL change
which addresses the flapping of back-to-back fiber connections.

Use miibus for SFP PHYs up to 1G copper. We retry in cases where the PHY is not directly reachable.
Set the correct IFM_100_SGMII flag when the phy speed has been set to 100. We remove xgbe_phy_start_aneg() since it's handled by miibus.

Add support for 100 and 1000 BASE-BX fiber modules

Add support for 25G multirate DACs which are capable of 10G.

While here, also fixup the LINK_ERR state. It was impossible to recover from this previously.

[bsdimp]

This change is too big. I'd suggest separating out things by each of the features that you call out in the commit message.
This will allow us to selectively back things out should there be trouble and allow users to bisect in case this breaks something.
I've not run tools/build/checkstyle9.pl on it, now it says:

ERROR: unnecessary whitespace before a quoted newline
#103: FILE: sys/dev/axgbe/xgbe-i2c.c:447:
+		axgbe_printf(1, "%s: i2c xfer ret %d abrt_source 0x%x \n", __func__,
ERROR: else should follow close brace '}'
#463: FILE: sys/dev/axgbe/xgbe-phy-v2.c:1294:
+	}
+	else if (sfp_base[XGBE_SFP_BASE_1GBE_CC] & XGBE_SFP_BASE_CC_PX)
ERROR: braces {} are necessary even for single statement blocks
#664: FILE: sys/dev/axgbe/xgbe-phy-v2.c:2930:
+		} else
+			xgbe_phy_rrc(pdata);

There's a few lines that are > 80 colmns, but are otherwise short enough.

[swhite2]

FWIW, this PR is already a subset (it pertains solely to link issues) of a larger diff, of which I've taken the liberty to set up for review in phabricator: https://reviews.freebsd.org/D40725. I can only assume the same logic you mentioned applies there, so if you'd like me to break it up into smaller changes here, I can abandon the diff in phabricator and start opening separate PRs.

[bsdimp]

Yes, the same logic applies... It might help get things reviewed, but also eases my concerns with too many things going in at once on the off chance there's an undetected issue (for hardware, it can be hard to get testers, or people really familiar with it). So if you break things up, I'll make sure they get into the tree and I'll try to beat the bushes to get reviewers).

[bsdimp]

I sent ahead and landed this change. I took some time to study it since I had thought it would be good to give better feedback than I had. I found most of the complexity was due to the large number of cases being handled due to the many different link types. While it could be broken down, I'm not sure there's good RoI on doing that. It seems close enough to me, and I'll keep an eye out for regressions in axgbe.

Sorry for taking so long to resolve this bit. It's hard to review changes for hardware you're unfamiliar with.

445bed5

[swhite2]

I appreciate you taking the time to review this. I agree the changes were too big to begin with, but FWIW most of these changes have been running stable in production for well over a year now.

These changes specifically target SFP(+) connectivity, which is a bit of a rabbit hole in terms of possible configurations. While I stayed away from other areas as much as possible, regressions for other connection types (RJ45, backplane etc.) cannot be ruled out - though I doubt adoption of this is very high, otherwise you would likely have seen more patches around this area as this driver needed quite some work to get into a stable state.

There are a few more changes here, I'll be sure to split these up for future PRs.