Configure new AP blank at times

[nilava]

Hey the page "Configure new AP" renders blank at times, after multiple refresh it loads, its usually happening when more than 7+ ssid are found. Is it a bug or is there any workaround for that or is it some issue with my code?

UPDATE :
Tried the library with the most basic sketch and still the same issue, so its an issue with the library i guess when there's more than 6-7 SSID are found.

Out of topic, is there any way to reset wifi credentials?

My Code - Gdrive Link

[ageurtse]

Reset wifi credentials, can be done by reflashing and clearing all flash storage.

As far as i know there is no simple reset command to erease all data.

@Hieromon, i need to load the wifi pag 2 times to see all networks.
Thought it was an issue from my side.

[nilava]

@ageurtse so i am not the only one with the issue it seems. Could you confirm with a simple sketch if the issue is still there? Obviously test when there's a lot of ssid!

[nilava]

I did further digging and found out the issue is there only when we load custom web page otherwise its absolutely fine handling lots of SSID.

[ageurtse]

I go try it tonight when back at home.
@nilava do you have some test files which i can use.
Vor non custom webpage i could use the basic example from the programm folder.
But what did you use for the custom webpage.
So if i have the same problem, hieromon could maybee reproduce the problem and solve it.

[nilava]

@ageurtse Look at your code for Wifi-Dimmer, just remove this line and try, you will see it works perfectly! As in without an custom webpage, unfortunately i couldn't come up with any workaround for customwebpage. From my guess there is no memory left after loading so many config and then after returning a huge string to be served.

In the library you can see that with lots of SSID the string "ssidList" can become really large and thus might lead to blank render of the config page.

AutoConnect/src/AutoConnectPage.cpp

Lines 1106 to 1126 in 2fee92c

	String AutoConnect::_token_LIST_SSID(PageArgument& args) {
	AC_UNUSED(args);
	String ssidList = String("");
	_hiddenSSIDCount = 0;
	WiFi.scanDelete();
	_scanCount = WiFi.scanNetworks(false, true);
	AC_DBG("%d network(s) found\n", (int)_scanCount);
	for (uint8_t i = 0; i < _scanCount; i++) {
	String ssid = WiFi.SSID(i);
	if (ssid.length() > 0) {
	ssidList += String(F("<input type=\"button\" onClick=\"document.getElementById('ssid').value=this.getAttribute('value');document.getElementById('passphrase').focus()\" value=\"")) + ssid + String("\">");
	ssidList += String(F("<label class=\"slist\">")) + String(AutoConnect::_toWiFiQuality(WiFi.RSSI(i))) + String(F("&#037;&ensp;Ch.")) + String(WiFi.channel(i)) + String(F("</label>"));
	if (WiFi.encryptionType(i) != ENC_TYPE_NONE)
	ssidList += String(F("<span class=\"img-lock\"></span>"));
	ssidList += String(F("<br>"));
	}
	else
	_hiddenSSIDCount++;
	}
	return ssidList;
	}

@Hieromon i hope you look into this.

[Hieromon]

@nilava I have not tested for reproducing with your uploaded code yet, but probably It will be insufficient memory as you guessed. The root cause is not the generation of the SSID list. It caused by the PageBuilder. PageBuilder concatenates the page contents by String.

https://github.com/Hieromon/PageBuilder/blob/3a6c5b27998ff2ddf601e27d50aa8cc2029fe13d/src/PageBuilder.cpp#L204-L227

So, the String class of ESP8266 reserves the heap by the realloc on concatenation. If realloc fails, String will be lost. The function is String::changeBuffer. This function has been updated with the upstream of ESP8266 core. esp8266/Arduino#5690
But, this PR does not seem to be included in stable release 2.5.0.

However, there is still the possibility that the heap will run out. AutoConnect generated HTML is long. If there are seven SSIDs, it will be near the 10K bytes. It means that a heap of about 20K bytes will be required for concatenation by PageBuilder. It is limit of an available heap that will remain after the ESP8266WebServer + AutoConnect + ArduinoJson resides excepts Blynk & ESP8266HTTPUpdateServer.
You can enable PageBuilder's debug option to actually check for out of memory. If the heap is run out, you will see the Serial message of the result of getFreeHeap.

PageBuilder.h L34

#define PB_DEBUG

https://github.com/Hieromon/PageBuilder/blob/3a6c5b27998ff2ddf601e27d50aa8cc2029fe13d/src/PageBuilder.h#L34

And I prepared a workaround for PageBuilder. Its workaround is to reserve the string buffer by the length of the content in advance and will suppress fragmentation of the heap that occurs each time a string is concatenated. I have tested it over and over. However, since its effectiveness can not be expected so much, I have not implemented the AutoConnect side yet.

https://github.com/Hieromon/PageBuilder/blob/3a6c5b27998ff2ddf601e27d50aa8cc2029fe13d/src/PageBuilder.cpp#L207-L214

Can you try with the debug option enabled on PageBuilder?
I will consider using AutoConnect's content string buffer reservation implementation with your code.

[nilava]

@Hieromon you were right, anyway i am happy with refreshing multiple times for now until there's a proper workaround for that.

Serial Output:
[AC] 5 network(s) found [PB] at leaving build: 18632 free [PB] Content lost, length:9211 [PB] Free heap:20536 [PB] Res:200, Chunked:1 [PB] Free heap:29760, content len.:0

[r3na]

Hey @Hieromon I tried the 0.98 branch and the bug is still there (in my case, here it is showing 30~40 networks ). Does this fix is already committed there?
Would not be possible also to limit the number of access points so to avoid this problem? Something like the 10th strongest ones.

[Hieromon]

@r3na

Does this fix is already committed there?

Sorry, not yet. You can try the buffer reservation option as an interim workaround. This option reserves a buffer for content building and will suppress fragmentation.

AutoConnect/src/AutoConnectDefs.h

Line 127 in 04c236f

#define AUTOCONNECT_CONTENTBUFFER_SIZE 0

However, if there are as many as 30 access points found, improvement can not be expected. HTML content length of the SSID list is about 250 bytes by each. The content length for displaying 30 SSIDs will exceed 15K bytes in total length, and it needs within an unfragmented area.

Would not be possible also to limit the number of access points so to avoid this problem?

It is possible. It equips with the Next Page button, and I will implement experimentally on the v.098 branch to reduce the displayed SSID to 5 at a time.
Thank you for your advice.

[Hieromon]

Hello @nilava, @r3na. I staged a trial version of paging the SSID list to the development branch. Please try this version in an environment where the esp8266 can find many SSIDs if you can.
The current configuration has 5 SSIDs for display per a page. You can change this with the AutoConnectDefs.h macro.

AutoConnect/src/AutoConnectDefs.h

Line 132 in 0fd3171

#define AUTOCONNECT_SSIDPAGEUNIT_LINES 5

[nilava]

Negative, didn't solve the issue, went down as low as 1 SSID to be displayed per page, didn't work still.
Thanks for your effort though!

[Hieromon]

@nilava, Thank you for testing. Hmmm, Yes. the operation is not stable. Sometimes appear, sometimes gone.
However, I will leave the option for paging display of the SSID list as a build option.
I think another way to resolve, sorry to keep you waiting.

[r3na]

@Hieromon just saw now the last updates. Let me know if you need any test from my side

[Hieromon]

Hello @nilava @r3na , I have changed the page generation and destruction scheme to avoid heap fragmentation and tried to chunk transfer some pages. The latest commit of the development branch absorbed these fixes.
Please try it from both the PageBuilder and AutoConnect development branches.

[nilava]

@Hieromon Amazing!!! 😍 Works Great! At times however the input form disappears, which is not a huge issue though, but as for blank page, it never comes anymore!

[nilava]

The minor bug i mentioned in the previous comment

[Hieromon]

@nilava Thank you for testing and report. It seems that the changed scheme is still insufficient. I will consider more to stabilize the behavior of the Config New operation.

[Hieromon]

@nilava I improved heap fragmentation in page building and staged the fix to the develop branch enhance/v098. However, I have not tested enough because I can not prepare an environment with 6 SSIDs on the same channel.
If the problem recurs, turn on debug logging with PageBuilder.h #define PB_DEBUG and check PageBuilder logs which say Failed building. If you find the failure log, please try increasing the following value.

AutoConnectDefs.h#L127

AutoConnect/src/AutoConnectDefs.h

Line 127 in 2c0207f

#define AUTOCONNECT_CONTENTBUFFER_SIZE (13 * 1024)

This value will reserve the content buffer at the time of the chunked transfer. PageBuilder cannot reserve the buffer in the first place if it is too large, and if it is too small, PageBuilder cannot create content. Maybe correct size scope is from 12 * 1024 up to 15 * 1024, I think.

[nilava]

#define AUTOCONNECT_CONTENTBUFFER_SIZE (15 * 1024)

This ensured maximum stability with as many as 12 SSID's! The issue is finally solved completely! Thanks! @Hieromon

[Hieromon]

@nilava Thank you for testing. The default value is (15 * 1024) based on your test results, which will be released in v098.