Sdcard in core

[jepler]

This pull request adds sdcardio.SDCard and, for select atmel-sam boards, sdio.SDCard.

sdcardio.SDCard is nearly, but not quite, compatible with adafruit_sdcard.SDCard: It takes a microcontroller.Pin for the chip select (cs) line, not a digitalio.DigitalInOut object, because this simplifies object lifetime and may help us to someday let vfs mounts survive soft-reset. It also defaults to a higher baudrate, 8MHz instead of 1.32MHz.

sdcardio benchmarks vary from card to card, but on one 4GB card, I measured an increase in read rates from 583 kB/s to 838.2kB/s, an increase of about 40%. Write rates remain very low, just 4kB/s(!) with adafruit_sdcard and sdcardio.

All sdcardio testing was on a PyGamer with built in SD card slot. Three SD cards were tested (4GB Sandisk, 8GB Sandisk, 64GB Samsung). Some lower capacity SD cards are on order and I'll verify they all work, or if they don't whether they work under adafruit_sdcard.

sdcardio is enabled on all "full-build"s except M0, PCA10100, and litex, for reasons outlined in the individual commits.

sdioio implements an SD card interface using the SDIO bus. This is enabled and tested on the Microchip SAME54 Xplained Pro board and is also enabled on the Grand Central. The relevant pins are on headers of the Grand Central M4 Express, but (likely due to dodgy wiring) did not work in my testing.

Known problems:

• If no card is detected, sdioio.SDCard hangs at initialization
• sdcardio.SDCard misidentifies the capacity of a particular 64GB card I have for testing, same as adafruit_sdcard

We made the decision to deprioritize the stm32f405 feather support for SDIO SD cards. This work is not finished and has not been renamed to suit the latest decisions, and so it have been removed from this PR. See https://github.com/jepler/circuitpython/tree/sdio-stm32-broken-rename -- We will get back to this at some point, but we would also love to see a community member pick up the work. Two main things remain to be done: Follow the naming convention of sdioio and finish the pin logic in the constructor.

[tannewt]

@dhalbert is this review on your radar?

[tannewt]

For now, let's call this _sdcardio just in the top level so that we can evolve the API as SDIO support is added. Internally the names can stay the same.

[dhalbert]

@tannewt Sorry, I looked for this the other day, and it seemed like a number of changes were pending, so I waited.

[jerryneedell]

OK -- I found the commit fixing the doc example -- thanks !

[dhalbert]

I only had stylistic comments. It looks good but I have not tested it. Are we going to rewrite adafruit_sdcard to be a thin wrapper around this for now, or just deprecate it?

[jepler]

Thanks @dhalbert I'll take your comments into account when I revise the PR to rename it as _sdcardio.

[jepler]

I think adafruit_sdcard will still be used for M0 devices where we don't have space for sdcardio in flash.

adafruit_sdcard could check for the existence of _sdcardio and use it instead if it's present. So there could be one version of various programs for both M0 and M4 boards.

[jerryneedell]

tested this PR on a grand_central_m4_express
It works!

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.3.0-187-g82024d078 on 2020-05-19; Adafruit Grand Central M4 Express with samd51p20
>>> import sdcardio_test
['hello.py', 'images']
>>> 

I had to make a few changes to the example since the Grand Central M4 does not use the statndar SPI bus for the SD Card. here is the example I ran

import adafruit_sdcard
import busio
import sdcardio
import digitalio
import board
import storage
import sys
import os

# Connect to the card and mount the filesystem.
spi = busio.SPI(board.SD_SCK, board.SD_MOSI, board.SD_MISO)
sdcard = sdcardio.SDCard(spi, board.SD_CS)
vfs = storage.VfsFat(sdcard)
storage.mount(vfs, "/sd")
print(os.listdir('/sd'))

took it one step further and actually executed a python script (hello.py) from the sdcard

Press any key to enter the REPL. Use CTRL-D to reload.
Adafruit CircuitPython 5.3.0-187-g82024d078 on 2020-05-19; Adafruit Grand Central M4 Express with samd51p20
>>> 
>>> 
>>> import sdcardio_lib
>>> import hello
Howdy
>>> 

here is the example sdcardio_lib.py

import adafruit_sdcard
import busio
import sdcardio
import digitalio
import board
import storage
import sys
import os

# Connect to the card and mount the filesystem.
spi = busio.SPI(board.SD_SCK, board.SD_MOSI, board.SD_MISO)
sdcard = sdcardio.SDCard(spi, board.SD_CS)
vfs = storage.VfsFat(sdcard)
storage.mount(vfs, "/sd")
sys.path.append("/sd")
sys.path.append("/sd/lib")

oops _I noticed that I was still importing adafruit_sdcard -- It still works when I remove it!

jerryneedell@jerryneedell-ubuntu-macmini:~/projects/grand_central$ cat sdcardio_test.py 
import busio
import sdcardio
import board
import storage
import sys
import os

# Connect to the card and mount the filesystem.
spi = busio.SPI(board.SD_SCK, board.SD_MOSI, board.SD_MISO)
sdcard = sdcardio.SDCard(spi, board.SD_CS)
vfs = storage.VfsFat(sdcard)
storage.mount(vfs, "/sd")
print(os.listdir('/sd'))

and

jerryneedell@jerryneedell-ubuntu-macmini:~/projects/grand_central$ cat sdcardio_lib.py 
import busio
import sdcardio
import board
import storage
import sys
import os

# Connect to the card and mount the filesystem.
spi = busio.SPI(board.SD_SCK, board.SD_MOSI, board.SD_MISO)
sdcard = sdcardio.SDCard(spi, board.SD_CS)
vfs = storage.VfsFat(sdcard)
storage.mount(vfs, "/sd")
sys.path.append("/sd")
sys.path.append("/sd/lib")

[dhalbert]

@jepler There are still some unresolved comments above (alphabetization and "not bitbangio.SPI")

[jepler]

This has been updated with SDIO for STM32F405 Feather.

[jerryneedell]

Tested SDIOCard example on feather_stm32f405_express -- worked fine!
Also tested adding /sd to path and executed simple script from the sdcard.

[jepler]

Note for myself: https://github.com/avrxml/asf/blame/61c7353e0183bbd0615db1d0a3144cbdcd09983a/common/components/memory/sd_mmc/sd_mmc.c

Also, this PR presently reverts two submodules to an intended ref. oof.

[tannewt]

I think it'd be better to separate the bus from the SDCard APIs here.

# Connect to the card and mount the filesystem.
bus = busio.SPI(board.SD_SCK, board.SD_MOSI, board.SD_MISO)
device = adafruit_bus_device.SPIDevice(bus, board.SD_CS) # Also takes frequency, polarity and phase.
# For SDIO
# device = sdioio.SDIO(clock=board.SDIO_CLOCK, command=board.SDIO_COMMAND, data=(board.SDIO_DATA,), frequency=25000000)
sdcard = sdcardio.SDCard(device)

[hierophect]

Just a couple comments about the Systick and peripherals structures. I'm going to take a closer look at HAL_GetTick and try to get a handle on what's going on with the low power code.

[jepler]

This has grown and is really multiple different topics. We need to split it up into multiple PRs. Our priority list becomes:

1. SPI SD card in core, supporting all platforms
2. SAM D5x/E5x SDIO
3. STM SDIO

(that's not to say we wouldn't love someone to pick up #3 before I get a chance to! It's still the only reasonable way to use the SD card on the STM32F405 Feather. The main thing that needs completion on STM32F405 is handling of pin assignments, instead of ignoring them as now)

We also need to nail down what the API is going to look like, to avoid more renames and incompatible changes if possible. We have discussed this informally a number of times, but I'd like to get to an official decision.

1. How many modules, 1 or 2? How are they named? Suggestion: sdcardio for SPI interface SD cards and sdioio (for SDIO interface) Conclusion: 2 modules
2. Are the module names underscored (_sdcardio) or not (sdcardio)? My preference: No underscore Conclusion: No underscore
3. Does the SPI interface take (spi: busio.SPI, cs: microcontroller.Pin) or (bus: adafruit_busdevice.spi_device.SPIDevice)? My preference: (spi, cs). One reason: we require that the SPI bus is a real hardware SPI bus, and that the CS is a real physical pin, while SPIDevice requires no such things. Conclusion: This is a good trade-off.

If you take my suggestions/preferences, this leads to the following code snippets for mounting SD storage, depending on interface:

sd = sdioio.SDCard(clock=board.SDIO_CLOCK, command=board.SDIO_COMMAND, data=board.SDIO_DATA)
vfs = storage.VfsFat(sd)
storage.mount(vfs, "/sd")

and

sd = sdcardio.SDCard(spi=board.SPI(), cs=board.SD_CS)
vfs = storage.VfsFat(sd)
storage.mount(vfs, "/sd")

We are going to do priorities 1 and 2 in this PR and push #3 to a separate PR or issue.

(comment updated based on discussion with @tannewt via discord)

[jepler]

@tannewt @dhalbert your comment on the above would be very welcome.

[jepler]

@tannewt @dhalbert I think this is ready for another round of review. The initial description of the PR has been heavily updated to reflect the current state of this branch.

[tannewt]

A couple minor things.

I believe I've addressed all your requested changes

[tannewt]

Thank you for this! Folks will be excited for the speedups!