Expose on-board I2C for accessories

[whitequark]

As suggested in https://github.com/whitequark/Glasgow/issues/67#issuecomment-429797935.

[marcan]

Do we want to do this / what would it look like?

We currently have I²C testpoints. Those could easily be replaced with a 3x2.54mm header (I2C+GND), but I think that may run up the wave soldering keepouts against some decoupling caps.

[marcan]

There's also a spot centered on the board, to the right of the TCAs, where this would fit vertically and not screw up the wave soldering more.

[whitequark]

I feel like this ought to be addressed systematically. Let's leave this out of C0; if someone wants digipots, they can use the testpoints. Once the need and a good path forward is clear, we can add this.

[marcan]

Yeah, we'd need supplies to make this actually useful (and depending on what the user is doing, that might mean 3V3 or VioA or VioB or even also 5V)

[marcan]

Just for reference since I already tried it, this is what it would look like with just SDA/SCL/GND (modulo fixing the silkscreen):

[whitequark]

Yeah, we'd need supplies to make this actually useful (and depending on what the user is doing, that might mean 3V3 or VioA or VioB or even also 5V)

Yes. And we can't extend J6/J7, since there's no space...

[whitequark]

My additional experience with I2C on revC0 indicates that any fault on the bus tends to bring the device to its knees until it's power cycled, and USB is not helping here a lot either. Therefore I do not think exposing I2C on revC or revD is a good idea. revE would probably have an I2C switch (or multiple) but that's quite a long time off.

[whitequark]

To be clear, zealous handling of I2C errors is something that's possible to do, but given that errors on the bus potentially affect DUT reliability (e.g.: your DUT can't handle voltages on its inputs while it's not powered, you unplug the DUT, but Glasgow has an I2C bus fault and doesn't shut down the LDO) I am very hesitant to go that route.

And even if we do handle all I2C errors, we'd still have to invest a lot of time into making sure every I2C device can be properly reset, the resets don't lead to glitches (this part is almost impossible), and we have a sound way to test this by fault injection, which is not something I think we can afford for revC/D either.

[marcan]

Strongly agree given what I've seen recently. Exposed I²C needs to be behind an isolator/mux.

That said, revD needs a mux anyway. I think it's feasible there, but it's not happening for revC.

[whitequark]

It's also kind of ridiculous that most I2C muxes are controlled via I2C. I think the only reliable way to use such a mux is to just reset it every single time before writing a new command byte to it, because otherwise you introduce timeouts--a software minefield--and latency--which can blow your deadlines.

[marcan]

Yeah, I was thinking about that too. This is only going to work if the mux has a reset line we can control, so we can unwedge it if a downstream peripheral is stupid. We probably don't have to worry much about sub-buses that we control, but we'd need to bracket any external I²C comms with mux resets to cleanly isolate external faults from internal peripherals.

[whitequark]

This is only going to work if the mux has a reset line we can control, so we can unwedge it if a downstream peripheral is stupid.

Right, but we don't have any pins on FX2 we can use for a mux reset line on revD, so I think this isn't happening for revD either. (We could technically put LDO enable lines under an I2C GPIO expander, but this is getting quite absurd at this point.)

[marcan]

Yes, that's a dealbreaker for revD. It's going to depend on how that road goes when we get there. I think there's still a good chance we decide we need more GPIOs anyway and end up using an expander, or something else. If things go in a direction where this is doable, why not do it. If no, then not.

[whitequark]

I am weakly against having such a feature on revD and not having it on revC for the following reason. revC and revD are supposed to be more or less equivalent, with the understanding that you trade off LVDS capability for double the I/O count. An adapter board should, ideally, work with both. That said, maybe I'll change my mind.

[whitequark]

Let's keep this open until we figure out for sure whether there's going to be a mux before revE.

[whitequark]

I'm thinking we could use the "aux" connector on revC to make the FPGA "proxy" I²C if it comes to that. But I still think that a solution that only uses the normal I/O bank pins is more robust...

[electroniceel]

(continuing from #117)
For an analog addon board, I'd like to have access to the internal I2C and 5V. Having 3V3 would also make sense since the I2C is using 3V3 levels.

I propose to use the space below the Sync-header for this. In the current revC design this place seems to be unused on most layers and the signals mentioned are already running near that place. The sync header would have to be replaced by a vertical 3x2 pin header. This way it could still be used with pin sockets like in the current design.

The addon board could route through the sync signal and provide it on a pin header again, so you could use the sync with the addon board installed.

---

About I2C muxes / isolators:

The problems you talk about appear when hotplugging something into the internal I2C bus, correct? Is hotplugging necessary? Wouldn't it be possible to adapt the spec of Glasgow to just allow plugging something in/out of the internal I2C when Glasgow is unpowered? Something like "the shrouded headers are secured against hotplugging & ESD, everything else is not".

You could of course add a mux with reset and so on to guard the internal I2C against influence from outside, but that will add extra cost and needs board space. Is hotplugging an addon board that important?

---

About controlling the addon board with some pins from the regular port a or b:

• I'd still need power, preferably 5V
• It makes it hard to properly autodetect if an addon board is installed and which. I'd have put an I2C EEPROM on the addon board which an applet could read and then automatically enable functions, hooks and so on for the addon board. Using regular io pins like this could interfere with the DUT
• The analog board I envision would just minimally interfere with regular Glasgow usage (maybe a few ohms series resistance, a few more parasitic pf). It routes through all of port A and B and I'd constantly leave it plugged into Glasgow. It just adds extra features like ADC on each pin or uses hooks into the regular applets to allow stuff like autodetecting voltages.

[whitequark]

The problems you talk about appear when hotplugging something into the internal I2C bus, correct? Is hotplugging necessary? Wouldn't it be possible to adapt the spec of Glasgow to just allow plugging something in/out of the internal I2C when Glasgow is unpowered? Something like "the shrouded headers are secured against for hotplugging & ESD, everything else is not".

No, it's not just hotplugging. Adding things on the internal I²C bus means this internal bus is more likely to misbehave. It is, in fact, very easy to make I²C misbehave, as anyone who's ever worked with it can tell you. Right now, we have a few carefully selected components on it, which are thoroughly tested, and so I'm quite certain that in normal operation it's not going to misbehave. (Consider how we thought we took good care of the added I²C level shifters in revC0, and how much breakage that caused.)

Glitches and strange behavior on the internal I²C bus mean two things.

1. If the FX2 cannot get a response from I²C in time, it (currently) hangs. It is possible to add timeouts to it, but AFAIK not all I²C fault conditions are recoverable. I think the fault condition where a device is stuck transmitting an ACK bit because it waits for the master to toggle clock, and the master is stuck because it thinks there's bus contention, is not recoverable with FX2.
2. If the FX2 is stuck, it will not be able to turn off Vio in case Vsense goes out of range. The voltage range alert function is supposed to protect a DUT from being powered via its digital pins in case the DUT power is lost but Glasgow still drives it. Right now, I'm confident that it will do its job in almost any circumstances (I've never seen it fail, at least) but with your change there is no confidence at all.

Robustness, especially robustness against operator error, is a core design principle of Glasgow and I will not compromise on it. In fact, if I had the option, I would not use I²C at all; the current use of I²C for a few carefully selected and tested chips on the board is a kind of compromise already, though it works well enough that I haven't had reasons to regret it. Unfortunately FX2 doesn't really leave any options here due to its low pin count, and there are no alternatives to FX2 in this corner of design space.

Exposing the internal I²C bus to arbitrary peripherals will reduce this robustness significantly and so it will not happen. An I²C mux that is reset out of band on every single I²C transaction is probably good enough (though it needs validation), but we physically lack the board space to fit one on revC, so it's not going to happen on that revision.

I propose to use the space below the Sync-header for this. In the current revC design this place seems to be unused on most layers and the signals mentioned are already running near that place. The sync header would have to be replaced by a vertical 3x2 pin header. This way it could still be used with pin sockets like in the current design.

I am especially opposed to this idea because it is very easy to miss the right row on the pin header when plugging in the sync cable and thus accidentally short the I²C bus to ground when sync activates. Or worse, 5V. That's never happening.

---

So is there a reason your addon board can't use the AUX connector? It would need power; I think we could fit a 4th pin with 3V3 there, but then it wouldn't nicely fall on the 0.1" grid, which is a bit sad. I don't think 5V there is going to happen because that's just one slipped screwdriver away from a fried board.

[whitequark]

I see these possible routes to addressing this:

1. Do nothing. Cons: no defined way to autodetect addon boards or communicate with them.
2. Explicitly define AUX as carrying I²C signals. Add pullups on it. Use the FPGA to proxy host I²C to AUX I²C, or maybe use gateware to drive AUX I²C directly. Cons: requires FPGA to be configured before the addon EEPROM can be read; does not protect FPGA from ESD or overvoltage at all; gateware complexity, potential timing problems; we lose AUX pins for other applications.
3. Add an I²C buffer with enable in the smallest reasonable package (I think PCA9306 fits between the level shifters), and route the enable to the FX2 so that it can enable that only while trying to communicate with the addon board. Cons: we absolutely have to use an I²C expander on revD now to control LDOs; restricts the addresses that can be used by addon boards (but I think 0x50 still works, and this is statically detectable anyway); we lose AUX pins for other applications.

Common for both 2 and 3 is the problem that only one addon board can use the AUX connector and it has to go over both of the ports.

Common for both 2 and 3 is the problem that if we expand AUX any further it stops being wave solderable.

@marcan What are your thoughts?

[whitequark]

3. I think PCA9306 fits between the level shifters

Of course, PCA9306 is another one of those awful I²C devices that come with severe restrictions on operating them that make them unsuitable for us. TCA9517 wouldn't fit. TCA9406 fits, but it has internal pullups that are disabled if OE is inactive, which rules it out for our application. TCA4311 seems ideal for our application but it wouldn't fit. PCA9515 wouldn't fit.

So that's pretty much the entire TI I²C buffer lineup ruled out for this application. This isn't going so well.

[whitequark]

@electroniceel So, the I²C switch exists (a regular analog switch can be used in this application if level translation is not required), but we couldn't find a connector configuration that would work well in this appliaction. Here's the IRC log.

I think revC1 is too far into the pipeline to do any kind of major change at this point, and there is no way to make this supported well without major changes. I think it is more important to support expandability well than to support it at all. The decisions we make right now (if we do), we're going to be stuck with them for the rest of eternity, like Arduino is now stuck with their misaligned connectors.

So, we will prototype something for revD0 and then, if it works out, backport it to revC2 or something. But there's no way to easily add this capability in a way that we can support forever, and it's too late to add it the hard way, so it's not going to be there in revC1.

[marcan]

@electroniceel So here's a proposal. We can't really come up with a "nice" official way of doing what you want on revC, but what we can do is make it easier to do unofficially. So: we're freeing up pin #⁠20 on the IO connectors (previously GND); this makes 19 and 20 both no connect. We will probably use those for something in the future, but for now, this gives you a total of 4 spare pins on the existing connectors. We'll also put I²C, 5V, 3V3 testpoints on the bottom of the board. This way, you can prototype your whole-Glasgow analog expansion board by just adding 4 fly-wires to the bottom of the board, assigning I²C and either or both supplies to the four extra pins on ports A and B combined. Then your board can just connect to the existing IO connectors, and can be easily connected and removed.

To be clear, we aren't going to implement this officially like this in any future revision, but at least this gives you a reasonable way to do it on revC1 unofficially, without requiring any extra hanging connectors or other horrible mods, just four wires on the bottom. This also allows you and others to experiment with other ideas, e.g. connecting I²C to both connectors (for standalone single-bank addons), or a supply to both connectors (for people who just want a big fat 5V to power something exceeding the Vio capability), or putting the AUX pins somewhere (for people who just need that one extra IO).

Does that sound useful for now?

[electroniceel]

I don't know of a suitable I2C switch that is small enough either. So I can understand you moving official support to the next board revision. The unconnected pins on the io header are ok for development and a prototype board.

If you plan on putting the testpoints below the sync header, you'd have space to use through hole ones. If you want to solder wires to them, through hole is much more sturdy. I don't care if they are filled with tin from wave soldering.

I suggest to add some silkscreen markers (for example on the back side) indicating the unconnected pins on the port headers. Not that someone expects ground there. Maybe extend that to printing the numbers, sense, Vio. I tend to forget the pinout and hate having to look it up all the time.

---

revD is planned to have the same fx2 and ice40, but more port IOs instead of LVDS, correct?
I guess you'd need more board space for that. Do you plan to completely redesign the physical layout or do you plan to keep some compatibility, like the distance and layout of the shrouded port headers?

---

I'm a bit sceptical if using '66 muxes in the next revision will help against the I2c logic of the fx2 locking up: if the glitch, emi,... happens while the autodetection code tries to search for an addon board, the fx2 may still lock up. So I think to solve this you'd need a full i2c switch with it's own i2c logic and an external reset, something like the PCA9546A (won't fit the current board of course).

I can fully understand your frustration with I2C. I had my share of lockup problems on projects I worked on. But if you are not keen on adding microcontrollers (see #115 (comment)) there aren't many alternatives left. I use open drain UART as bus on several of my projects and that works well. But all members are microcontrollers and it is just single-master.

---

I don't think 5V there is going to happen because that's just one slipped screwdriver away from a
fried board.

You already have a 5V test pin exposed, also the legs of the components. A screwdriver, metal scrap or whatever there can fry your board too. If you want to properly protect a bare board against that, you'd need thick conformal coating everywhere. Don't think that makes sense.

Just use a case that encloses the whole board and just leaves the port headers and usb exposed. 3D printed, laser cut acrylic, whatever.

I need 5V for the analog muxes. Also an addon board may need some switching converters to create additional power rails. It is not efficient or easy to run them from 3V3. Especially during startup they can draw a lot of current to charge the output caps. The 3V3 may dip from this and that could cause stability issues somewhere else on Glasgow. A complex power sequence logic is not what you want.

[whitequark]

revD is planned to have the same fx2 and ice40, but more port IOs instead of LVDS, correct?

We'd need to go to the HX8K in CB256 package as there aren't enough FPGA balls for four I/O banks.

I'm a bit sceptical if using '66 muxes in the next revision will help against the I2c logic of the fx2 locking up: if the glitch, emi,... happens while the autodetection code tries to search for an addon board, the fx2 may still lock up. So I think to solve this you'd need a full i2c switch with it's own i2c logic and an external reset, something like the PCA9546A (won't fit the current board of course).

I agree that isolating the internal I²C devices completely would be more reliable than connecting the I²C on addon board to the main bus. That pretty much rules out exposed I²C on revC.

But if you are not keen on adding microcontrollers (see #115 (comment)) there aren't many alternatives left.

Correct. But I think "no autodetection" is also a perfectly reasonable approach. It's not that important.

I use open drain UART as bus on several of my projects and that works well. But all members are microcontrollers and it is just single-master.

I've considered open-drain UART, but the FX2 has neither spare pins for it nor processing power to bitbang UART in software at any reasonable baud rate. So it's not a great solution.

You already have a 5V test pin exposed, also the legs of the components. A screwdriver, metal scrap or whatever there can fry your board too. If you want to properly protect a bare board against that, you'd need thick conformal coating everywhere. Don't think that makes sense.

You misunderstood. If the AUX connector is exposed to addon boards, it will have a corresponding opening in the case, so even if you have a board that is enclosed, it's still very easy to short it. If it is not exposed to addon boards, having 5V on it is about as bad as having it on the test points (i.e. not very bad), but it's also pointless for the purposes of addon boards.

Also, I'd be totally behind a thick conformal coating if I knew of a way to economically apply it while masking all the sensitive areas, like the connectors for customizing pull resistors. I don't think that exists, though?

A complex power sequence logic is not what you want.

We have a power sequencer that will take care of this. The FX2 has it as one of its requirements, though I've never seen anyone else actually use one.

I need 5V for the analog muxes. Also an addon board may need some switching converters to create additional power rails. It is not efficient or easy to run them from 3V3. Especially during startup they can draw a lot of current to charge the output caps.

This is true, but keep in mind that I've never planned for addon boards that would be active independent of Vio. All of the addon boards I was thinking about are things like passive adapters to IC sockets, serializers/deserializers, ADCs or DACs that are directly controlled by the applet, and so on.

And for good reason! As this issue nicely demonstrates, supporting such active addon boards well adds a massive amount of complexity (we haven't even touched software yet...). It's hard enough to make a purely digital multitool. Adding plug-and-play analog addons to it is nightmarish, and I don't believe I can pull it off at all without a lot of iteration--and the first iteration that will be shipped to the public is not it.

[whitequark]

If the AUX connector is exposed to addon boards, it will have a corresponding opening in the case, so even if you have a board that is enclosed, it's still very easy to short it. If it is not exposed to addon boards, having 5V on it is about as bad as having it on the test points (i.e. not very bad), but it's also pointless for the purposes of addon boards.

Oh by the way, I am very much opposed to having always-active 5V on any pin of the IO bank connectors in any board revision. Absent-mindedly plugging a jumper wire into a pin next to the one I actually need, or to the wrong side of the connector because I'm looking at it the wrong way, is something I do constantly. Any such power pins will be on a separate connector next to the actual I/O bank.

[electroniceel]

You misunderstood. If the AUX connector is exposed to addon boards, it will have a corresponding opening in the case, so even if you have a board that is enclosed, it's still very easy to short it. If it is not exposed to addon boards, having 5V on it is about as bad as having it on the test points (i.e. not very bad), but it's also pointless for the purposes of addon boards.

Hmm. That all depends on the kind of case concept & design you have in mind.

If you have one case that covers Glasgow itself and addon boards connect through openings in this case, then you are right. But then the addon board is unprotected and can still be fried by touching metal parts. So the addon board would have to come with it's own case to protect itself.

So why not expect the addon board to provide a case that encloses Glasgow and the addon board together instead? In small volumes and 3d printing or laser cutting the price difference would be negligible. Then you wouldn't need openings in the case for the addon connectors.

Also, I'd be totally behind a thick conformal coating if I knew of a way to economically apply it while masking all the sensitive areas, like the connectors for customizing pull resistors. I don't think that exists, though?

All conformal coating stuff I've seen being done was completely manual, with a brush or syringe-style applicator. That is expensive. But the biggest problem I see is that if you use proper coating, you can't remove it ever again. So it completely prevents replacing broken components, expansion, hacking, development,... - I think the intended audience for Glasgow values having these options.

[electroniceel]

Oh by the way, I am very much opposed to having always-active 5V on any pin of the IO bank connectors in any board revision.

I concur. They shouldn't be on io banks.

[whitequark]

So why not expect the addon board to provide a case that encloses Glasgow and the addon board together instead? In small volumes and 3d printing or laser cutting the price difference would be negligible. Then you wouldn't need openings in the case for the addon connectors.

I see your point, but I've never considered this kind of addon boards in the first place. Nothing in Glasgow is designed to support this use case well. Also, I can't really imagine any addon boards that would always want to be in the middle and between anything else you might connect, other than your analog board.

[electroniceel]

Also, I can't really imagine any addon boards that would always want to be in the middle and between anything else you might connect, other than your analog board.

Ah, you mean the combined case makes changing addons more complicated? You have a point there.

Maybe the top part of the regular case could be made to easily snap in. Then the addon-case could be made to snap into the same position instead. Don't think that works with the laser cut acrylic designs I know of. Would probably mean 3d printing.

Another idea would be to have the regular Glasgow case have an opening for the addon connectors, but they are protected by a snap-on cover if not in use.

I think there are possible solutions for this, but the board design doesn't really depend on having a complete case design done. I'd just go forward with a case that protects the addon connectors. Providing users with a STL of a new case with access to the connectors is much more simple than to add connectors to a board that is already shipped without...

[whitequark]

As described above, we can't physically fit a suitable I²C mux on revC and it's not reliable enough with a simple analog switch. So there isn't going to be an addon connector...

[electroniceel]

I know. Was just thinking ahead for revD.

[whitequark]

I think for revD we can provide a level translating mux (I think there are muxes that are explicitly ESD rated) and additional protection for any other purposes. I imagine I²C could be provided directly at the I/O bank as the passive adapters would then go for the easier solution—there's a ton of EEPROMs that work anywhere from 1V8 to 5V—and the active adapters would make use of the additional 5V power.

[electroniceel]

Good idea. An EEPROM for autodetecting passive addons like socket adapters would make sense to automatically use the correct pin mapping.