Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Direct SD Card support #93

Closed
luc-github opened this issue May 16, 2016 · 36 comments
Closed

Direct SD Card support #93

luc-github opened this issue May 16, 2016 · 36 comments

Comments

@luc-github
Copy link
Owner

As suggested by @openhardwarecoza : add direct wires to SD Card reader for file upload

SD Library examples suggest SD card attached to SPI bus as follows:
** MOSI - pin 11
** MISO - pin 12
** CLK - pin 13
** CS - pin 4

@luc-github
Copy link
Owner Author

currently I use wifi SDCard with my ESP, would be great to not handle 2 devices.
Issue would be to avoid concurent access to card, so need a flag in board fw to let esp know it can use sd card and same for esp to release SD card to board

@luc-github
Copy link
Owner Author

luc-github commented May 16, 2016

On AZSMZ board with Smoothieware connected to PC, the SDCard is automatically visible under windows as a Removable Disk - using Repetier and so M28/M29 to upload file does not work -

11:31:27.257 : No firmware response detected for sd action. Assuming success and continue.

The file does not show in windows explorer

@luc-github
Copy link
Owner Author

luc-github commented May 16, 2016

After restarting board the file is present on SD Card - so concurrent access seems working
Edit : looks random - not always success - so not reliable

@luc-github
Copy link
Owner Author

luc-github commented May 16, 2016

Seems depending of how fast is repetier to load files before windows mount the removable disk.

From ESP the M20 command never work when SDCard is mounted on window when in repetier is seems working but actually it do not - I guess repetier host cache the information instead of really getting them and so giving wrong files list

@ghost
Copy link

ghost commented May 16, 2016

Going to tag in @arthurwolf - arthur, this is some very beta craziness,
but we are attempting to do the crazyness of accessing the sd on a multi
master spi bus (; off the esp8266. Or something like that. I honestly
missed a lot of the conversation between Ray and Luc too (;
On 16 May 2016 13:01, "Luc" notifications@github.com wrote:

Seems depending of how fast is repetier to load files before windows mount
the removable disk.

From ESP the M20 command never work when SDCard is mounted on window when
in repetier is seems working but actually it do not - I guess it cache the
information instead of really getting them and so giving wrong files list


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#93 (comment)

@arthurwolf
Copy link

I didn't know about this project, it's very very nice.

Liam Jackson is actually working on something that looks like this, essentially teaching an ESP to do the following : 

  1. Allow users to connect to the board via serial at a fixed baudrate ( with a telnet port )
  2. Get it's Wifi config from the board via serial ( "cat" command http://smoothieware.org/console-commands )
  3. Essentially "act" exactly like a Smoothieboard, so anything coded for Smothieboard's Ethernet also works with it, which includes :
    a) When doing an http get for "/", answer with a static webpage
    b) When doing an http get for "/sd/something", use the "cat" command to get the file and answer it's content
    c) When doing an http post for "/command" with the command as a post arg, call that command over serial, and answer it's answer over http

Because the Smoothieboard is supposed to come soon with a web interface on it's SD card, implementing a pass-through like that to that interface, would probably be a good idea.

@ghost
Copy link

ghost commented May 16, 2016

Now thats what I was looking for!
On 16 May 2016 13:20, "Arthur Wolf" notifications@github.com wrote:

I didn't know about this project, it's very very nice.

Liam Jackson is actually working on something that looks like this,
essentially teaching an ESP to do the following :

  1. Allow users to connect to the board via serial at a fixed baudrate (
    with a telnet port )
  2. Get it's Wifi config from the board via serial ( "cat" command
    http://smoothieware.org/console-commands )
  3. Essentially "act" exactly like a Smoothieboard, so anything coded for
    Smothieboard's Ethernet also works with it, which includes :
    a) When doing an http get for "/", answer with a static webpage
    b) When doing an http get for "/sd/something", use the "cat" command to
    get the file and answer it's content
    c) When doing an http post for "/command" with the command as a post arg,
    call that command over serial, and answer it's answer over http

Because the Smoothieboard is supposed to come soon with a web interface on
it's SD card, implementing a pass-through like that to that interface,
would probably be a good idea.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#93 (comment)

@luc-github
Copy link
Owner Author

well current version support the console-commands from TCP/IP terminal like putty and from web interface - but on web interface I did not implemented the full output yet - currently only info/error/warning are displayed but I planned to add full output on an additional list

image

image

@arthurwolf
Copy link

Wow, really really nice.

On Mon, May 16, 2016 at 3:09 PM, Luc notifications@github.com wrote:

well current version support the console-commands from TCP/IP terminal
like putty and from web interface - but on web interface I did not
implemented the full output yet - currently only info/error/warning are
displayed but I planned to add full output on an additional list

[image: image]
https://cloud.githubusercontent.com/assets/8822552/15290294/d10efbc2-1b77-11e6-80ce-651374daccd9.png

[image: image]
https://cloud.githubusercontent.com/assets/8822552/15290347/2726ca62-1b78-11e6-9c52-77ac34e33689.png


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#93 (comment)

Courage et bonne humeur.

@luc-github
Copy link
Owner Author

the web UI has few hook to retrieve specific information in json format (sd files/temperatures/positions/etc..) because it is easier, but it is mostly a pass-through so any serial command will work on web ui - and on any TCP/IP port defined in setting page

but where web ui has authentication - TCP/IP port currently do not have authentication like telnet

@luc-github
Copy link
Owner Author

but back to original topic - all SD access go through serial pipe instead of direct SD reader connection with ESP. which is not really fast for file upload

@luc-github
Copy link
Owner Author

luc-github commented Aug 5, 2016

Web upload to SD Card using ESP give me 95000byte/s which is a lot faster than serial transfer
but seems I got instability if SDCard reader is powered by ESP and if ESP is using also SPIFFS - need to investigate - if some of the issue are due to too much pwer drained by card and/or if there are some conflict in core as SD and SPIFFS share same API naming, SPIFFS has a namespace but not SD (I use #define FS_NO_GLOBALS) I see some update in git to limit the objects - I will test it

I have damaged 2 ESP using external power as may not used proper circuit 😞 orz

@ghost
Copy link

ghost commented Aug 5, 2016

I have been playing with this too, and can confirm that its not power
related. I get lockups and hangs with sd when using the arduino sd card
library along with webserver. (even without spiffs) - but man, that
upload speeds are sooo nice, its worth spending time to get this working
On 5 Aug 2016 15:40, "Luc" notifications@github.com wrote:

Web upload to SD Card using ESP give me 95000byte/s which is a lot faster
than serial transfer
but seems I got instability if SDCard reader is powered by ESP and if ESP
is using also SPIFFS - need to investigate - if some of the issue are due
to too much pwer drained by card and/or if there are some conflict in core
as SD and SPIFFS share same API naming, SPIFFS has a namespace but not SD
(I use #define FS_NO_GLOBALS) I see some update in git to limit the objects

  • I will test it


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#93 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHVr2xdsdf_aLHyU2BUED95ETMumVDh-ks5qcz1RgaJpZM4IfFdO
.

@luc-github
Copy link
Owner Author

luc-github commented Aug 5, 2016

Actually I got 0 issue when doing my test with SDWebserver I was able to upload 9Mb file, several times when I was not able to go over 1.5Mb with my FW - may be was luck , but I was using external power for SD card

@ghost
Copy link

ghost commented Aug 5, 2016

Try downloading the file you just uploaded
On 5 Aug 2016 16:18, "Luc" notifications@github.com wrote:

Actually I got 0 issue when doing my test with SDWebserver I was able to
upload 9Mb file, several times when I was not able to go over 1.5Mb with my
FW - may be was luck


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#93 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHVr20C68Owu9KbjIB3Q8g9QNFpwY4i_ks5qc0YagaJpZM4IfFdO
.

@luc-github
Copy link
Owner Author

ok thank you will test that, I did only one way actually

@ghost
Copy link

ghost commented Aug 5, 2016

I only played for an hour or so earlier in the week. Didnt get a
resounding confident feel. But hey, i am watching your work very
closely. Sd proxy to smoothie was smoothesp's goal (but no progress in a
few months) , and i bet we can raise you a few donations to get this
feature working! Shout where i can help!
On 5 Aug 2016 16:32, "Luc" notifications@github.com wrote:

ok thank you will test that, I did only one way actually


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#93 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHVr2_0jtwsG7jNdjIDROY6VIY0bbJcJks5qc0mPgaJpZM4IfFdO
.

@luc-github
Copy link
Owner Author

I did several tests not during 1h but several minutes :
Upload 7Mb file - 3 times - then download it 3 times, then compare downloaded file with original one using meld which said it is identical. and watch the serial output when during test so far so good

I got few : LmacRxBlk:1, which means too many data are coming on wifi, but this is because I am in debug mode and seems no data corruption as final result is ok

Can you share what was your circuit and what size of file did you used ? I would like to reproduce the issue you meet

Here my system using 2 usb adapters one for ESP and one for SD Card reader:
image

with https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WebServer/examples/SDWebServer

@ghost
Copy link

ghost commented Aug 5, 2016

Ahh yes, lmacrxblk was what i was getting too. Several, eventually
responses botg serial and wifi suspends till reset.

My test setup is a esp12-e, ad over spi. I tried running a http <->
serial loop while uploading (genuine use case: i want machine feedback
while uploading.)
On 5 Aug 2016 18:12, "Luc" notifications@github.com wrote:

I did several tests not during 1h but several minutes :
Upload 7Mb file - 3 times - then download it 3 times, then compare
downloaded file with original one using meld which said it is identical.
and watch the serial output when during test so far so good

I got few : LmacRxBlk:1, which means too many data are coming on wifi, but
this is because I am in debug mode and seems no data corruption as final
result is ok

Can you share what was your circuit and what size of file did you used ? I
would like to reproduce the issue you meet

Here my system using 2 usb adapters one for ESP and one for SD Card reader:
[image: image]
https://cloud.githubusercontent.com/assets/8822552/17442798/e5da8990-5b37-11e6-80a9-3ae367808f5a.png

with https://github.com/esp8266/Arduino/tree/master/libraries/
ESP8266WebServer/examples/SDWebServer


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#93 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHVr25cCZThh9GJfE8jvcwCNCnwf3FWLks5qc2DggaJpZM4IfFdO
.

@luc-github
Copy link
Owner Author

luc-github commented Aug 5, 2016

I see - webserver is not what you must use if you want to get feedback while uploading as it is a SyncServer
I suggest you the Asyncwebser from me-no-dev - it is in my long todo list to integrate it
#110

and to limit the lmacrxblk you need to use 160Mhz and not default 80Mhz CPU frequency

@ghost
Copy link

ghost commented Aug 5, 2016

Cool!
On 5 Aug 2016 18:30, "Luc" notifications@github.com wrote:

I see - webserver is not what you must use if you want to get feedback
while uploading as it is a SyncServer
I suggest you the Asyncwebser from me-no-dev - it is in my long todo list
to integrate it
#110 #110


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#93 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHVr29hRDwLRsaVImnb32Vvzh6ysjmpiks5qc2UOgaJpZM4IfFdO
.

@luc-github
Copy link
Owner Author

Ok now ESP3D is working well (so far) using SD Card - still need to work on several feature like automount and File manager for SD card and blocking other requests when doing file upload to avoid to overload wifi but on good way finally

I also found an issue and reported it : github.com/esp8266/Arduino/issues/2368 but I can workaround easily for the moment

@luc-github
Copy link
Owner Author

what do you mean ? wires follow normal SD reader recommendation for ESP as mentioned in first thread:
image
after it depend of your sd reader

@Aleks-MO
Copy link

Aleks-MO commented Oct 15, 2016

Sorri
** MOSI - 13
** MISO - 10
At esp12f - 22 pins.

@luc-github
Copy link
Owner Author

what is that? it show 404 page not found

@Aleks-MO
Copy link

Aleks-MO commented Oct 15, 2016

@luc-github
Copy link
Owner Author

you mixed pin number and GPIO number check page 7
6 IO12 GPIO12; HSPI_MISO
7 IO13 GPIO13; HSPI_MOSI; UART0_CTS

@Aleks-MO
Copy link

Aleks-MO commented Oct 15, 2016

thanks, understood the SPI bus.
** CLK - 5 IO14 GPIO14; HSPI_CLK
** CS - 16 IO15 GPIO15; MTDO; HSPICS; UART0_RTS

@luc-github
Copy link
Owner Author

Support is ready and will be available later so I close issue

@GriffinPaquette
Copy link

Hi guys. I have designed a small attachable device that has an ESP-12E with a micro sd card reader directly on the back of it. I have attached the pins as noted above, but it does not seem to see the sd attached to the back. It does see the one on the mainboard through its serial connection, but as file upload, that way is incredibly slow, I want to upload through the direct connection over SPI. Is there something that I need to change from the current version of the ESP3D compilation? I just flashed the most current source code.

The board I am using is Smoothieware based.

expansion board

Really appreciate the help guys!

@DouglasPearless
Copy link

DouglasPearless commented Jul 26, 2018 via email

@GriffinPaquette
Copy link

Hmm, this was exactly my fear, Douglas. I appreciate the detailed write-up nonetheless! Is direct SD access for Smoothie still out of the picture for general users in the short term? ESP3D has quickly become my goto for a quick and easy web UI for my boards, but can't stand the slow upload over serial.

@DouglasPearless
Copy link

DouglasPearless commented Jul 26, 2018 via email

@GriffinPaquette
Copy link

I know that Roy from Panucatt has a modified version of ESP3D that has direct SD access for his X5 Mini Wifi. It really interests me to have a cheap onboard wifi module especially with the ease of use that ESP3D has.

As far as I can tell there is an intermediate buffer that interrupts the SPI bus from the LPC in some way. Makes sense but dunno how exactly it works. What I do know is that the upload speed is quite excellent (Nice work Luc).

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants