Compilation of w600 port IV

[rkompass]

Checks

• I agree to follow the MicroPython Code of Conduct to ensure a safe and respectful space for everyone.

• I've searched for existing issues regarding this feature, and didn't find any.

Description

We continue here the discussions
Compilation of w600 port and
Compilation of w600 port II - II (all closed now)
of development/debugging the w60x port in MP branch w60x.

Code Size

No response

Implementation

• I intend to implement this feature and would submit a Pull Request if desirable.
• I hope the MicroPython maintainers or community will implement this feature.
• I would like to Sponsor development of this feature.

[rkompass]

I don't know whether you are interested to take this up:
I found my modified machine/machine_spi.c from last year. It was about 80% complete iirc.
The intent was to remove (with compiler flag) the option to use the (from datsheet)

6.2 High speed SPI device controller
Compatible with the universal SPI physical layer protocol, the host supports high-speed
access to the device through a data format that interacts with the host. The maximum
supported operating frequency is 50Mbps.
-Compatible with the general SPI protocol;
-Selectable level interrupt signal；
-Supports up to 50Mbps rate;
-Simple frame format, full hardware resolution and DMA

and restrict to

6.13 Master/slave SPI controller
Synchronous SPI master-slave function is supported. Its working clock is the system internal
bus clock. Its characteristics are as follows:

• The transmit and receive paths each have a FIFO of 8 words deep;
• Master supports 4 formats of Motorola SPI (CPOL, CPHA), TI timing, macrowire timing;
• The slave supports four formats of the Motorola SPI (CPOL, CPHA);
• Support full duplex and half duplex;
• The master device supports bit transmission and supports up to 65535bit transmission.
• The slave device supports transmission modes of various length bytes;
• The maximum clock frequency of the slave device input SPI_Clk is 1/6 of the system clock;

to free the relevant (now permanently occupied) RAM area.
I can collect all the relevant information if you are interested in working on this (collaborating, of course).
Although I have to admit I'm not really familiar with that any longer now.
I got distracted by the possiblity to use the websocket then and for that started to study Javascript and so on...

machine_spi.zip

[robert-hh]

Sorry for the tardy reply. I just noticed that message. Yes, you had asked that before. Sine slave mode is not used by any other port, and HSPI seems to be used only for slave mode, it could be dropped. Looking into the SDK, there seems no options for HSPI master mode.

Note that the file which you attached seems to be an old version. The actual machine_spi.c is way smaller after removing some code duplication.

[robert-hh]

Scanning through the source files of the WM_SDK and the port, it seems that the HSPI/SDIO is using 8k of RAM. The note in wm_hspi.h which talks of 26k seems wrong. The addresses mentioned there are wrong as well. At the moment we have:

20037fff: Top of Python heap
20038000: 8 k DMA data structures (not sure if these are needed)
2003A000: 8 k for HSPI and SDIO
2003C000: WiFi buffers.

[rkompass]

Yes. These are in SDK/Include/wm_ram_config.h :

#define MASTER_SPI_DMA_ADDR     0x20038000UL  /*Master SPI use buffer when spi use dma transfer mode*/
#define SLAVE_HSPI_SDIO_ADDR    0x2003A000UL  /*High speed SPI or SDIO buffer to exchange data*/
#define WIFI_MEM_START_ADDR     0x2003C000UL  /*Wi-Fi use buffer to exchange data*/

So the 8k of HSPI can be saved and hopefully also the 8k DMA.

I will now re-read our old conversation about RAM and perhaps re-post here.

[rkompass]

The rest is about the FreeRTOS tasks which you had optimized afterwards. Then we had the firmware update change.

[rkompass]

I try to understand my changes to machine_spi.c:

1. I introduced a new preprocessor switch, aiming at the ability to the abolish hspi option from the start.

#if MICROPY_HSPI_SLAVE
#include "wm_hspi.h"
#endif

if it is 0 then all aof wm_hspi.h is not included.

2. I put the info about spi type in the switch

#if MICROPY_HSPI_SLAVE
    uint8_t spi_type;/* 0-lspi, 1-hspi */
#endif

and later switched out all the code for the hspi.

[rkompass]

The section

       if (-1 == self->bits) {
            tls_spi_trans_type(SPI_DMA_TRANSFER);
        } else {
            if (8 == self->bits) {
                tls_spi_trans_type(SPI_BYTE_TRANSFER);
            } else if (32 == self->bits) {
                tls_spi_trans_type(SPI_WORD_TRANSFER);
            } else {
                tls_spi_trans_type(SPI_DMA_TRANSFER);
            }
        }

of machine_spi_init() alone determines whether DMA is used.
I self->bits , which defaults to DEFAULT_SPI_BITS, which is 8, is 8 or 32 then no DMA is used.
Can we exclude that the bits setting -1 or any other is used?

[robert-hh]

Can we exclude that the bits setting -1 or any other is used?

No, we cannot exclude that. And we cannot exclude that people use the slave mode. Maybe the buffers could be dynamically allocated, or the reserved space could be reduced.

[rkompass]

Scanning through the source files of the WM_SDK and the port, it seems that the HSPI/SDIO is using 8k of RAM. The note in wm_hspi.h which talks of 26k seems wrong. The addresses mentioned there are wrong as well.

That might indicate that the HSPI/SDIO functionality is never used, because, if it were, it would be in conflict with the others.
So a simple fist approach might be to remove the 8k Buffer for that and then test SPI and memory.
I can try to rework the new machine_spi.c in the same spirit like the old one.
That would ensure that HSPI/SDIO functionality is not used.

What would one use for testing SPI reliably? Does an SDcard attached externally using sdcard.py suffice for that?

At present we have (in wm_hostspi.h):
#define SPI_DMA_BUF_MAX_SIZE 8160
#define SPI_DMA_MAX_TRANS_SIZE 4092

which also could/should be reduced, of course.
I wonder if not the first should be the double of the second at least.

Also there is #define SPI_USE_DMA in wm_hostspi.h
and quite some code in wm_hostspi.c is made dependent on that.
We might try to compile setting that to 0 and testing SPI afterwards.

I thing this whole thing makes sense thinking of a later w800 compile, which should be quite identical. As all register definitions are the same and the SDK is literally a copy of the w600. With another processor and the addition of Bluetooth.

[rkompass]

I'm testing now SPI for SD card and for that read the MP Micropython user guide.

There to my surprise I found:

There are some components in WM_SDK which is useless for MicroPython. These components can be cut off
before compiling to reduce this code size. Users can open the WM_SDK/Include/wm_config.h and modify the
macro of such useless component from “CFG_ON” to “CFG_OFF”.
The suggested useless components are:

#define TLS_CONFIG_HS_SPI CFG_OFF
#define TLS_CONFIG_HOSTIF CFG_OFF
#define TLS_CONFIG_RMMS CFG_OFF
#define TLS_CONFIG_HTTP_CLIENT CFG_OFF
#define TLS_CONFIG_NTP CFG_OFF

On your MD page describing this port you only mention the latter 4 components.
Was there a reason why you did not recommend
#define TLS_CONFIG_HS_SPI CFG_OFF ?
This is exactly what we need now.
in the SDK it is still
#define TLS_CONFIG_HS_SPI CFG_ON /*High Speed SPI*/

[rkompass]

I tested the SPI with a 4GB SDCard. Using mendenms new driver and test.
Had to compile FatFS in for that.

Driver and test files are here:
sdcard_mendem.zip

Hardware SPI at 20MHz seems to work well.

The test yielded:

------- results with spi baudrate=2_000_000 -----------

Filesystem check
[]

Multiple block read/write
5400 bytes written
11 bytes written
5405411 bytes written at  0.1058971 MB/s
final file size 5405411 expected 5405411 read 5405411 rate= 0.1465636

Single block read/write
11 bytes written
11 bytes read

Verifying data read back
Small file Pass
Big read Pass

Tests passed

------- results with spi baudrate=20_000_000 -----------

Filesystem check
['rats.txt', 'rats1.txt']

Multiple block read/write
5400 bytes written
11 bytes written
5405411 bytes written at  0.1840578 MB/s
final file size 5405411 expected 5405411 read 5405411 rate= 0.3213871

Single block read/write
11 bytes written
11 bytes read

Verifying data read back
Small file Pass
Big read Pass

Tests passed

Change to the higher SPI speed really yielded faster data transfer. For comparison mendenm reported

Speed is still slow, but for a lot of data-logging stuff, maybe OK. At 120 MHz CPU, 40 MHz SPI on my Pi Pico, I get about 600kB/s write speed, and just over 1000 KB/s read speed. Much of this may due to small block sizes and poor buffer handling.

Given we have only 80MHz CPU and 20MHz SPI our values are quite o.k..

Now testing changes of SPI can begin.

[robert-hh]

The file README.md is a mix of the initial version by Winner and my additions, which start in the section "Building the firmware". I do not think that alone setting #define TLS_CONFIG_HS_SPI CFG_OFF will help, since the MicroPython API is still there, referring to HS_SPI functions.

Thank you for testing with the SD card. I had tried an SD card earlier and an external SPI chip as well and found both working. I made no speed test, but speed may not something to be expected with the W60x.

[rkompass]

Good morning Robert,

I made changes to machine_spi.c similar to the changes in the last version of that file and compiled.
==> spi did not work (at least) not with my SDCard.
Then I re-set #define TLS_CONFIG_HS_SPI to CFG_ON again, and now spi works (with my changes).
The new machine_spi.c is here:
machine_spi.zip

So if you like to try that out...
I think we can afterwards abolish the HSPI 8k buffer.
But it would be good to understand why the #define TLS_CONFIG_HS_SPI unsetting did not work.
The internals are not so clearly separated as they should, it seems.#
Do you have an idea?

[rkompass]

I do not think that alone setting #define TLS_CONFIG_HS_SPI CFG_OFF will help, since the MicroPython API is still there, referring to HS_SPI functions.

Confirmed that as it did not compile afterwards. Compiler errors lead me to apply the same changes I had done half a year ago. Now there is the #define MICROPY_HSPI_SLAVE introduced, but I leave that to you to find the right place for it (in the Makefile?).

[robert-hh]

With TLS_CONFIG_HS_SPI set fo CFG_ON tls_hspi_init() is called in wm_main.c. And that has some calls to the WiFi section to an ATCMD app. Hard to tell what is affected and if that is needed. Maybe that's a left-over from the initial application of the W60x as a WiFi modem.

#define MICROPY_HSPI_SLAVE must be in the Makefile if link options are directly affected, which seems to be the case if RAM is to be extended. Since you have the set-up with SPI working, you could try whether you can swap the DMA and HSPI RAM blocks.

[rkompass]

I inserted MICROPY_HSPI_SLAVE = 0
into the Makefile.

The MP user manual tells:

In MicroPython, when the ID is -1, the software SPI function can be used, and when ID is 0, the
hardware SPI can be used.

Now the machine_spi.c internals tell me that there is the self->spi_type that is 0 for ordinary SPI and 1 for HSPI.
These are also taken from the MP SPI id.
This would imply that you could (with MICROPY_HSPI_SLAVE = 1) instantiate a SPI object with id=1 which uses HSPI.
Related question: Did you ever test SoftSPI on the w600 (id=-1)?

[robert-hh]

The general rule for MP SPI seems to be, the ID >= 0 is a hard SPI, and id=-1 is SoftSPI as legacy option, to be dropped in the future. The option -1 is not supported for the W60X port and not for other ports like RP2, STM32, MIMXRT, etc.
The variant id >= 0 is port-specific. For W60x, indeed id=0 is the ordinary master mode SPI, and id=1 is the slave mode HSPI.
So the supplied ID in machine_spi_make_new() should be checked for:

0: Use ordinary SPI.
1: USE HSPI, if supported
any other value: Raise an error.

[robert-hh]

There is a line:
#define MICROPY_PY_MACHINE_SPI_MAKE_NEW machine_hard_spi_make_new
in mpconfigport.h which seems not to be used at all.

[rkompass]

I reverted to the complete build including HSPI and tested:
With id=-1 an SPI is instantiated which reports id=0 afterwards. And it behaves identically.
So I confirm there is no SoftSPI, at least not using the id=-1 (but which is stated in the old user manual).
Was soft SPI taken out previously in reworking the w600 port?

With id=1 (HSPI) the SDCard test program hangs, which is expected given that only slave SPI is possible).

So either we re-establish SoftSPI or leave the id out completely, as it has no function.
For compatibility the id=0 should be tested an give an error message (with id=-1) that SoftSPI is not available (or available as SoftSPI) and with ids greater than 0 also report an error that higher SPI instances are not available.
Edit:
I see now that is exactly what you proposed.

[rkompass]

With MP version 1.10 there was SoftSPI available. Using id=-1 it reported then as SoftSPI.
I will try wheter it worked then.

[robert-hh]

May not be necessary, since all major ports dropped that option already. And no one complained so far.

[rkompass]

O.k., I was able to do a

spi = SPI(-1, baudrate=400000, sck=Pin(Pin.PB_16), miso=Pin(Pin.PB_17), mosi=Pin(Pin.PB_18))  # for older w600 versions
spi.init()                               # Ensure right baudrate
sd = sdcard.SDCard(spi, Pin(Pin.PB_15))

and the SD card was correctly reported as v1 or v2 with MP 1.10 (e.g.[SDCard] v1 card), but at mounting there were errors.
So we'll also drop SoftSPI.
So then only the check of SPI id and error report is needed.

[robert-hh]

There is still machine.SoftSPI. But getting SoftSPI with id=-1 can be dropped.

[rkompass]

machine.SoftSPI works well. At a much slower transfer rate (baudrate=400000) the transer is 1/16 for write, 1/30 for read of that with full speed hard SPI.

[rkompass]

Compiled with TLS_CONFIG_HS_SPI set fo CFG_OFF in addition to the now standard MICROPY_HSPI_SLAVE = 0
the initialization of the SDCard fails with a timeout.
This is strange and after inspecting the files I can only assume that either:

1. tls_hspi_init() does something that is needed beyond HSPI function, or
2. the HSPI FreeRTOS task created therein somehow is used by ordinary SPI.

A strange thing is that the binaries created with or without TLS_CONFIG_HS_SPI set are exactly the same size.
That would be an argument to just set it to CFG_ON, but I'm hesitating.

[robert-hh]

I found that after changing parts of the SDK I had to do a full clean with make clean and re-build everything including the SDK files.

Another question. Besides setting MICROPY_HSPI_SLAVE = 0 in Makefile, did you set the compile flag like

ifeq ($(MICROPY_HSPI_SLAVE),1)
CFLAGS += -DMICROPY_HSPI_SLAVE=1
endif

And have the conditional #if MICROPY_HSPI_SLAVE in the C code? Not that it affects your last test, but for enabling it would be required.

[rkompass]

No, of course not, as I'm a neewbee in this regard.
I thought that Makefile variables automatically would be translated into compiler definitions.
But seeing the above now it makes much sense!

[rkompass]

I discovered

#define SPI_USE_DMA in wm_hostspi.h. It's used for some compiler switchery setting up the DMA.
Adding a 0 after it deactivates these switches, is that right?
I tried with #define SPI_USE_DMA 0 and #define TLS_CONFIG_HS_SPI CFG_OFF and the SDCard is not recognized.

Compiled with #define SPI_USE_DMA 0 but #define TLS_CONFIG_HS_SPI CFG_ON the SDCard test passes with full speed (like with 20MHz baudrate, which it has now).

So this could be the way to get rid of both the HS SPI 8k buffer as well as the 8k DMA buffer.
But still I would prefer to have it working with the #define TLS_CONFIG_HS_SPI CFG_OFF too before proceeding.

[robert-hh]

Adding a 0 after it deactivates these switches, is that right?

Depends on how the switch is used:
#if SPI_USE_DMA is false if SPI_USE_DMA is 0 not not defined, #ifdef SPI_USE_DMA is false is not defined, but true for #define SPI_USE_DMA 0.

With #define SPI_USE_DMA 0 and #define TLS_CONFIG_HS_SPI CFG_OFF simple hardware SPI should still work for any use including SD card. Otherwise it's not a feasible option.

[robert-hh]

Found it in w60x/modules/modhashlib.c. It seems that the internals of mp_obj_malloc_var() had changed.

Edit: The culprit was most likely commit 58e2b3d18 2024-02-25 w60x: Replace m_new_obj...() with mp_obj_malloc() and fix socket.send().

[rkompass]

I made a short search: This is the only occurrence of mp_obj_malloc_var() in the w60x port.
Thanks for the last commit.

[robert-hh]

I did the same search before pushing the commit.

[rkompass]

Unfortunately Webrepl still does not work as last year.
Now you can login, but at the prompt there is no echo.
A few seconds after the >>> prompt Disconnected appears.

At the w600 a dupterm: Exception in read() method, deactivating: OSError: 0 is reported.

[robert-hh]

No problem here. But it may be a different topic. Try to open a USB session first until you get a REPL prompt, and then close it again.
I see the behavior that main.py does not seem to start until I open the terminal session.

[robert-hh]

At the Wemos and Thingsturn boards, RTS of the USB/Serial bridge is connected to Reset of the W600. At the Wemos and newer Thingsturn boards by a 100nF cap, at older Thingsturn boards direct. These need USB to be started once to boot the w600. I have a Wavgat 602 here without these connections. I can power it from a wallwart USB power supply, and it accepts and runs a Webrepl session just fine.
Edit: If I power the Wemos w600 from a wallwart USB power supply, it just boots up and accepts Webrepl sessions. And this power supply has definitely only power at the cable. I know it because I opened it once for repair.

[rkompass]

Thank you. I'll try again later.

[robert-hh]

I took care that WIFi is up and connected before starting webrepl at the board. webrepl_setup puts the start of webrepl into boot.py. That may be before WiFi is connected.

[rkompass]

I started webrepl from the command line. I have WIFI setup in boot.py. Perhaps I shall setup webrepl in main.py.

[rkompass]

WebREPL connection from: ('192.168.178.46', 39412)
1+2
3
>>> dupterm: Exception in read() method, deactivating: OSError: 0

This time I was very fast: I managed to enter my password and afterwards even entered 1+2 + enter.
The Disconnected appears after about 5-6 seconds always.

[robert-hh]

Strange. Which board do you use? I tried 3 different board, and all work reliable. Maybe the WiFi connection dropped, or a power supply issue.

[rkompass]

I used Wemos W600-PICO.
With MicroPython v1.20 everything works o.k. if I start webrepl from screen terminal. (Starting from Thonny ignores the input., o.k.).
So the script/browser (Brave) combo is o.k..
I'll compile myself now and try with the latest MP version with the same board.
Do you happen to have an older firmware? I'd like to save that 1.20 version somehow.

[rkompass]

Same problem with this board, from screen. Everything like with MP 1.20 but webrepl disconnects after about 7 seconds.
I used your build wm_w600_WEMOS_W600.fls.

[rkompass]

I built my own image and now the
dupterm: Exception in read() method, deactivating: OSError: 0 appears after about 1 second.
No time to enter a password.

[rkompass]

Here is my image, if you-d like to try. I edited the Makefile so that secboot is 0 and FatFS is 1.
wm_w600.zip

[robert-hh]

I used the image from my web site and that one is working well too. I tried your image and that one works as well. No timeout. No disconnect.

[rkompass]

That is a mystery now.

[robert-hh]

Which browser do you use? I use Firefox and the latest Debian stable Linux.

[robert-hh]

Here the communication locks up after about 10 minutes of silence with the message at the board:
dupterm: Exception in write() method, deactivating: OSError: [Errno 113] EHOSTUNREACH
So somehow the wifi connection is lost / the board locks up.

[robert-hh]

I started another test with a Thingsturn board, where I've cut the connections between RTS and reset and pulled reset to 3V3 by a 12k resistor. At all other boards there is not external pull-up for reset. RunnIng now for >1 hour.

[rkompass]

Which browser do you use? I use Firefox and the latest Debian stable Linux.

Brave on Manjaro.

Perhaps dupterm should retry on small read/write errors.?

But I rather think there is another regression present.

[robert-hh]

Perhaps dupterm should retry on small read/write errors.?

I do not expect that to be useful. The IP stream should have it's own transport layer reliability.

But I rather think there is another regression present.

Hard to tell without being able to repeat it. The last test is running now stable >2 hours.

[rkompass]

O.k., my webrepl is here:
webrepl.zip
I built for the W600_EVB_V2 and flashed there. Same issue: dupterm: Exception in read() method, deactivating: OSError: 0
after 4 seconds. Just enough to enter the password and get a >>> prompt.
Tested with firefox.

Next idea: I will start WIFI as AP

I found the problem: in my boot.py there was the w600.ftp started -----------------

with that commented out, webrepl runs fine.
Sorry for troubling you.

[robert-hh]

Sounds fine. I had two boards here running fine since 5 and 7 hours. At the one Wemos W600 which stopped after 10 minutes before I added a strong pul-up at reset to 3V3 (56 Ohm). The other board had alreaday a pull-up. Maybe that helps as well. I'll try tomorrow again w/o the 56Ohm pull-up.

[rkompass]

Now even vshymanskys viper-ide starts with the w600. Although it has an error as soon as one wants to use the filemanager at the left. But its webrepl works now and the files are displayed at the left...

[robert-hh]

The two w600 here are connected and running now since 16 and 18 hours. So I consider them as stable.
Edit: The Wemos runs as well stable without the pull-up at Reset.

[rkompass]

I was puzzled because your w600 tests of the web_repl with viper-ide were succesful.
I now found (and reported) that that depended on the browser.
With Firefox I could do all the file operations too. With Chromium only a simple directory display and repl worked.
File operations lead to malfunction.

[robert-hh]

I tried Google Chrome as well and it works fine. Maybe it's a problem of Brave.
Update to the most recent version of Viper IDE. It works better than the one 2 days ago.

[rkompass]

Just commented at viper-ide github.