How to detect ENC28J60 module "hang" state

[zbx-sadman]

I use ATmega328-based Arduino & ENC28J60 (revision 6) & Arduino_UIP (fix_errata12) in my hobby project for a long time. My device(s) handles requests from the monitoring service (Zabbix). But i can't achieve stable operation of the device. It can work two weeks on the production (noisy) network and hang then, or work only 2...3 hours and hang again. Also it can work for a month on the home network. I'll try to change power supply, set additional capacitors, give more RAM (~250 bytes free in the work state) and so... I have no luck in all cases - activity led is blink, but no any data in the ethClient.read()

Nevertheless, any attempts to hang module by hand was failed too ;) I have used flood-attacks without success - the module stops to answer for a few time and works again when attack is off.

After that i have add ENC28J60 registers state output in my sketch, connect device to noisy network and began to wait. When module is freeze, i have connected to the UART and saw that ECON1=00000000b.

ECON1 contain the RXEN bit : Receive Enable bit. If its 1 - packets which pass the current filter configuration will be written into the receive buffer. 0 - All packets received will be ignored.

Seems that cleared RXEN bit is the reason for such behavior - recieving no data with activity led blinking.

In the Arduino_UIP code, i did not find the RXEN bit cleaning operations. Now i'll try to call Enc28J60.init() when 0 == ECON1.RXEN.

I hope that the information was helpful And may be someone will found workaround to get more stability with ENC28J60.

[prezeskk]

Could you write a sample code to detect this situation?
In my project ENC28J60 hang too

[zbx-sadman]

It's seems as dirty hack at this time.

I changed utility\Enc28J60Network.h file - move readReg() sub from private to public members area.

Then i use this pieces of the code:

 #include <UIPEthernet.h>
 #define NET_ENC28J60_ECON1                                0x1F
 #define NET_ENC28J60_ECON1_RXEN                           0x04
 #define NET_ENC28J60_CHECK_PERIOD                         5000UL  // 5 sec
  ...
 setup() {
  ...
  }

 loop () {
  uint32_t prevENCCheckTime;
  while (true) {
    uint32_t nowTime = millis();
    if (NET_ENC28J60_CHECK_PERIOD <= (uint32_t) (nowTime - prevENCCheckTime)) {
        // Enc28J60 is Enc28J60Network class that defined in Enc28J60Network.h
        // ENC28J60 ignore all incoming packets if ECON1.RXEN is not set
        if  (! (Enc28J60.readReg((uint8_t) NET_ENC28J60_ECON1) & NET_ENC28J60_ECON1_RXEN)) {
            Serial.println ("ENC28J60 reinit");
            Enc28J60.init(netConfig->macAddress); 
        }  
        prevENCCheckTime = nowTime;
     }
     ...
     various_network_procedures();
     ...
   }
}

[prezeskk]

Thanks

[gwest7]

@zbx-sadman did you flood the data to or from the ENC28J60? I can send a lot of data to the module, but when the module sends just a few bytes it hangs.

I also tested the init code above, but it had no improvement.

[gwest7]

Are you still using your module as described above or have you perhaps found a stabler library for the ENC?

[zbx-sadman]

Yes, i've flood module with TCP and UDP traffic by hping3 --flood ip-address utility. Pinging is stops (ping ip-address in another terminal session) when flood was started and resume on flood breaking. fix_errata12 branch is used. And i has flood my sketch with a continuous flow of request for ~2 days - no hangs anyway.

I've found additional "signs" on hanged ENC module: ESTAT.BUFFER and ENC. EIR.RXERIF will be raised. It seems that sometime strange interference just bump the network chip. Maybe it's all the same electrostatic interference. I don't know. IP and MAC shows on debug output as it should be, but ENC module's registry say to me that error occured ... My ENC hanged on ~second week after restart and therefore code debugging is pain.

I still use Arduino_UIP because its have the same commands that stock Ethernet library and allow to use both Ethernet shields with the same code.

[prezeskk]

Here https://github.com/TMRh20/arduino_uip/tree/fix_errata12 is fork of this library with some changes

[gwest7]

@prezeskk are you running the fix_etrata12 fork with no lockups? I'm busy stress testing the fork with zbx-sadman's init and will let it run through the night.

[zbx-sadman]

@gwest7, may be yours sketch have a memory leak? In my experiments i found out that low memory will stops network functions working. If 250 bytes remain free after compiling (by IDE report) - device stops to reply on ping. May be compiler does not calculate correctly memory allocating and some classes from included libs eat the memory on init... But for my sketch it is a fact - no ping reply if <250b memory is still free. I think that this situation require to deeper source code analyze, but i not great programmer and haven't any debug tools :(

[gwest7]

Ok, I'll check my sketch again because it did freeze up again during the night. The fix_etrar12 firk is definitely stabler. Thank you for the help!

[zbx-sadman]

So, i has read datasheet and internet again. Network chip vendor recommend to use TTL level shifter on MISO (i don't use INT) for using ENC28J60 with 5V MCU. As might be expected - on my module from Aliexpress this shifter is not installed (but my device worked...)

And, on the Microchip forum, some participants are recommend to install the additional 100 ohm resistors (depicted as a blue rectangle) on other SPI bus lines, if the ENC is connected to 5V MCU.

I can't imagine how it protects the RX buffer in my case (and can't rewire my dev board at this time to testing), but may be it helps to you.

[gwest7]

Mine is connected to an Atmega328 at 5v soldered to a project board. Will make these changes too. Thanks for the effort!

[GunterO]

Maybe I'm suggesting here things you've already tried, but my lockups went away when I didn't use the 3.3v power supply of the Arduino board anymore, and used a 5v->3.3V step down converter to power the board.
My ENC28J60 board wasn't capable of connecting directly to 5V, and the 3.3v supply of the Arduino is not sufficient (even it "apears" to work). In that case you don't need to use a TTL level converter for the control lines.

[zbx-sadman]

@GunterO, yes using 3.3v from Arduino board is a bad idea )
Nevertheless, my module have onboard LM1117 LDO Regulator, but RX-error problem is exist anyway.

I agree - if network module and MCU board is powered from the same supply, voltage dropping on both regulators must be equal or close and ENC's TTL level must be enough for MCU, but who knows how good voltage regulator from Aliexpress ;) ENC can work on 3.1V (from datasheet), but it not enough voltage for 5V MCU input.

So, i think it's worth a try level converter. May be it did not help, but all will be wired as vendor recommend, and we can exclude this possible hardware error.

[zbx-sadman]

I still fight with my ENC.

Today i wrote multithread app on perl to make more test of my device. This script just fork process that opened connection to device, send request (~15 bytes) and recieve answer. And no more. Device just wait ~1sec before send answer and close connection.

I supervised the chip's registers EPKTCNT, EIR, and some others. I've found that EIR.RXERIF bit will be raised on two session, but my device still active and answers the request. EPKTCNT registry shows to me ~22-23pkts. Microchip's datasheet says to me followng: When a packet is being received and the receive buffer runs completely out of space, or EPKTCNT is 255 and cannot be incremented, the packet being received will be aborted (permanently lost) and the EIR.RXERIF bit will be set to '1'. Once set, RXERIF can only be cleared by the host controller or by a Reset condition. And its true - RXERIF do not clear when i stop perl script. And device still working.... i did increase UIP_CONF_BUFFER_SIZE to 200 - no any luck, all the same.

It turns out that this small traffic (15 bytes request every seconds in two thread) are cause the ENC buffer overflow? But device still answer and have no dropped requests. It make me crazy.

Now i want to test new strategy: I will monitor the time elapsed since the last successful incoming connection, and if detect no activity on one minute or so - check ECON1.RXEN && EIR.RXERIF bits. If first is dropped and second is set - just make Enc28J60.init(...). It allow to reinit chip not so often to make ping of device without lost packets

I know - it looks like mad action, but i see no other ways to avoid hard hangs at this time.