Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE REQUEST] Modifying the graph to allow the long-term data logging #539

Closed
CZvacko opened this issue Jan 28, 2023 · 166 comments
Closed

[FEATURE REQUEST] Modifying the graph to allow the long-term data logging #539

CZvacko opened this issue Jan 28, 2023 · 166 comments

Comments

@CZvacko
Copy link
Contributor

CZvacko commented Jan 28, 2023

If the user collects values over a long period of time (weeks, months) and with a short interval (150s), then working with the chart is quite slow. Any zooming or other operation takes a long time, for example for data collected for 2 weeks (50000 rows).

Maybe if the application could split the data by weeks (and allow to select a week using a drop-down list when displaying), it would be better and could collect data even for a whole year.

Currently I have to reset the datalog every 2 weeks.
@CZvacko CZvacko changed the title Modifying the graph to allow the long-term data logging [FEATURE REQUEST] Modifying the graph to allow the long-term data logging Jan 28, 2023
@fredlcore
Copy link
Owner

Hm, I was 100% sure that we had an open ticket for this request already, but I can't find it right now, so I'll leave this open until I find the other one. Otherwise, we'll continue here. In short: Yes, that would be a very handy feature, but since there is no fully working SQL(ite) or other kind of database available on the Due/ESP32, it's difficult to implement this. If someone has ideas how to effectively purge data older than a certain timeframe, I'm happy to hear more about it.

@CZvacko
Copy link
Contributor Author

CZvacko commented Jan 28, 2023

The values are currently read from a file, not from memory, right? My idea was to write a separate file for each week, e.g. datalog_w01.txt, datalog_w02.txt. Then just plot the desired file (datalog_w01.txt ...), no need to have a dymamic selection for the date range.
Maybe some users will prefer to keep the current way, then some setting could be added to allow the user to choose if they want a single log file or separated by weeks.
Also, there should be an option to download all log files at once.

@fredlcore
Copy link
Owner

Yes, they are read from a file. Separating the files in weeks is not too feasible, I think, because then you can't see for example what happens from Sunday over to Monday in one graph. Usually time intervals are more intuitive and could easily be adjusted to the users' needs (7 days, 1 month etc.). The problem is the "maintenance work" that would have to be done at a given time where old log entries are discarded. In SQL(ite3) that's really easy and fast, but there is no full implementation for microcontrollers yet.

@CZvacko
Copy link
Contributor Author

CZvacko commented Jan 30, 2023

Yes, sql would be ideal, but still better separate weekly logs than nothing. Concerning logging over the next week,, if someone will expect it in advance, then he can switch the (proposed) setting to use continuous logging.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 17, 2023

The JavaScript libraries used for /DG displays (C3+D3) are indeed slow when dealing with larger datasets. The older /DG implementation (using D3 only) might be faster. You can use it instead of the C3+D3 version by preceding #define USE_ADVANCED_PLOT_LOG_FILE in BSB_LAN_config.h with //.

One easy (?) hack with the C3+D3 version might be to filter the /D data in its JavaScript right after reading, e.g. by applying a regular expression passed in with the url, e.g. /DG?DateFilterRegEx=1.\.02\.2023 or something like that for 10.-19. Feb 2023 only. However, this seems hardly user friendly. Providing a more usable way to specify the time frame for filtering is conceivable, but would require more work.

My personal solution to this problem has been to keep log intervals long and/or log periods short. :)

@fredlcore
Copy link
Owner

Hm, the filter is actually quite a nice idea, because if we use dropdowns or whatever for setting the timeframe, the corresponding regexes could be created from that setting. However, it would still require to always read the complete dataset from the SD-card...

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

Yes, but that size penalty would hurt only once, not for working with the graph (zoom, pan, ...), and also not for its initial display.

Interactive filter settings could take different forms, e.g. show only the most recent n lines, show only data between dates x and y (possibly with y=now as a default), using every tenth value only, ... Since /D shows dates as dd.mm.yyyy and not yyyy-mm-dd, deciding if a date lies between x and y isn't as simple as it could be; for that, I'd (suggest to) transform the current format into the latter one before doing the actual filtering.

Do you know if there's enough people using /DG to display large datasets to warrant the implementation effort? So far I've thought that /DG is not meant to replace a proper FHEM system's graphing capabilities.

@fredlcore
Copy link
Owner

Of course it would hurt for its initial display because you would still have to load the complete data set even though if you only want to display the data for last week. That's where a SQLite implementation would significantly speed things up.

As for FHEM: If you have ever worked with FHEM, you'll know that its graphing capabilities is one of its worst qualities. FHEM's strength lies in fine-grained controlling of functions which is something that BSB-LAN cannot reproduce. The functionality of /DG as it is now is not really usable unless the user takes regular action (i.e. regularly deleting/moving data). Having a function that is half-baked does not make sense, so we either have to improve it or remove it. And yes, there are quite a few people using it.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

That's a bit like saying: all cars that cannot carry a ton of bricks in their trunk doing 200 km/h don't make sense, or all cars that need to be refueled are not really usable. I consider the current /DG implementation very useful, and much more versatile than cars, to stick to that example: It gives you both detailed (speed) and long period (load capacity) graphs - and if you're patient or willing to clear the data (refuel) every once in a while, even both at the same time. ;)

I'll take a lot at extending the current /DG implementation to interactively filter data at the client side, if that's desired.
Doing the filtering on the server side (SQLite, ...) or providing a completely different /DG implementation is something I don't see in my capacity.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

If/when filtering on the client side is done: The data in /D is highly redundant and would compress very well, to reduce transmission time. Of course, server side filtering would bei better, though.

@fredlcore
Copy link
Owner

The older I get, the less I think that analogies are helpful in any way because people tend to discuss on the details of the analogy more than they should do on the actual matter.
I have found implementations of SQLite that could work with BSB-LAN, but it's written for the IDF environment, so I guess it needs patching to work with the Arduino IDE which we use in this project:
https://github.com/siara-cc/esp32-idf-sqlite3/tree/c34beba7318fc00953b33db215b10dd23cb31924

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

Sqlite3.c is 7.36 MB. Sounds pretty big to me for an arduino/esp32 library used for what, to my understanding, is not bsb-lan's core functionality.

@fredlcore
Copy link
Owner

You know that the size of the source code (including in this case hundreds of lines of comments) says nothing about the size of the compiled binary?

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

I've seen enough code to know that, as a general rule, there is some correlation between source code size and binary code size. And I've seen enough code to be skeptical about fitting SQLite3 functionality in just a few KB. The size given here sounds more reasonable to me - or at least that order of magnitude.

The current /DG implementation accounts for less than 2 KB of bsb-lan's binary code, iirc.

@fredlcore
Copy link
Owner

So you seriously compare a full-blown SQLite installation on a regular PC with an adaptation with reduced functionality specifically designed for a microcontroller?

If you want me to take your argument seriously, provide relevant, fact-based information, for example after compiling one of the examples and see how much memory it consumes. This is what I will eventually do once I have spare time. Otherwise this discussion leads us nowhere.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

On my system, the sqlite3_spiffs example from https://github.com/siara-cc/esp32_arduino_sqlite3_lib uses 624653 bytes for the program and 22004 for global variables.

The ESP32_Console example from https://github.com/siara-cc/sqlite_micro_logger_arduino uses 376505 / 22288 bytes.

The BareMinimum example that comes with the Arduino IDE uses 211053 / 16048 bytes.

Subtracting the BareMinimum values from the SQLite / MicroLogger examples' values leaves us with 413600 / 5956 and 165452 / 6240 bytes, respectively.

Both 404 KB and 162 KB are the same order of magnitude as 500 KB.
q.e.d. ;)

@fredlcore
Copy link
Owner

Thanks. Have you deducted the boilerplate that comes with every esp32 binary? That alone is already several dozens of kilobyte and already part of the BSB-LAN binary, so it would not be added when using sqlite.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 18, 2023

Yes, see above.

@fredlcore
Copy link
Owner

Ah, ok, noted. Adding one of the SD_MMC examples (that's what will be used for SD card logging) to BSB-LAN's code adds 366kB to flash memory and 5kB of used RAM. While 500kB would have more or less maxed out the available flash in standard ESP32 configuration running BSB-LAN, 366kB still leaves quite some air to breathe.
The Micro Logger library is not an option as it does not (yet) support the VACUUM command to actually shrink the size of a database after deleting expired rows.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 19, 2023

I've implemented client side filtering to alleviate /DG performance issues when dealing with large datasets:
grafik
Two date input fields (from ... to) to narrow down data to be plotted. When loading /DG, they are set to the min/max dates found in the dataset (/D) - unless there's >10k lines of data, in which case the 'from' date is set to where the last 10k lines of data begin.

To signal when not all available dates are selected for plotting, a '!' is put next to the date input field that's cutting off data (both in the screenshot here). Changing one of the date input fields redraws the graph.

Compared to server side filtering (using SQLite or whatever), this solution still wastes time by both transmitting the whole /D dataset for /DG, and filtering it each time the plot is redrawn. However, what takes up the bulk of the time is highly reduced (unless the user sets the from/to dates to include all data): the actual plotting of thousands of values.

I've tested this implementation using Firefox 109.0.1 and Chromium 110.0.5481.100 on Ubuntu 22.04.1, as well as on my cheap, old Android phone with Chrome.

The code change adds 544 Bytes to the bsb-lan binary's size.

Let me know if I should upload this /DG version to github!

@DE-cr
Copy link
Contributor

DE-cr commented Feb 19, 2023

P.S. I've tested this on my i5-3320M / 2.6 GHz laptop using a data set of >300k data points and found the performance quite good - unless I set the from/to date filter to plot more than about 30k data points at once.

@fredlcore
Copy link
Owner

fredlcore commented Feb 19, 2023 •
edited
Loading

Cool, thanks!
544 Bytes sounds great of course, how long did it take to display let's say a week worth of data in this 300k dataset? I assume the file would then be around 30MB (or that order of magnitude ;) )?

EDIT: I'm not sure if I remember correctly, but I think you are not using an SD card in your setting? That is the main problem that data throughput of the SD card is rather slow (on the Arduino it was several dozens of kB/s). So downloading a 30MB file to display would take 300 seconds at 100kB/s. That was the point of departure for me to think of a server-side solution that in itself does not have to parse the whole file for just a subsection because SQL(ite) would know where to look for the relevnt data sets.

@fredlcore
Copy link
Owner

We could then think of a URL-command that purges the dataset except for the last x days of data (worst case by copying over the remaining data points from the old data file to a new file, removing the old and renaming the new file).

@DE-cr
Copy link
Contributor

DE-cr commented Feb 19, 2023

Times on my system (see above), using Firefox, data set = 31 days * 24 hours * 60 minutes * 7 parameters = 312480 data points:

  • 1104 ms to plot one day (10080 data points in 7 lines),
  • 3848 ms for one week,
  • 15932 ms for the whole month.

The above times are for plotting only, not including the /D data transfer from server to client. I've not done repeated timings for the data subsets above, only a single timing each.

I've done the above timing tests w/o including bsb-lan communication, loading a local file from my computer instead. Since I've created that file artificially (and because I'm lazy), it is smaller than actual /D output and weighs in at 14 MB. A typical line in this file looks like this:
20230131235900000;2023-01-31 23:59:00;7;7;120;
instead of something like
364593010;01.05.2022 00:00:15;8314;Kesselrücklauftemperatur Ist;66.7;°C.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 19, 2023

Btw, I don't see myself running into /DG performance issues anytime soon, even with the existing implementation: I currently log two parameters only, once a day. :)

@DE-cr
Copy link
Contributor

DE-cr commented Feb 19, 2023

Yes, my ESP32 doesn't have an SD card.

@fredlcore
Copy link
Owner

fredlcore commented Feb 20, 2023 •
edited
Loading

Thanks, I didn't see the rendering on the local PC or mobile phone as a problem and your findings support that assumption, although I'm a bit surprised that rendering a month takes 15 seconds. For comparison: I log 49 values every five minutes for 28 days (i.e. approx. 395.000 datapoints) into a SQLite database on my Synology NAS (which runs on a mediocre Intel processor) and have a PHP script displaying these datapoints in 14 different graphs, and to dynamically generate these 14 graphs with all the data contained on one page takes 6-7 seconds, from pressing the reload button to the page finishing loading.
But even though it's 50% faster, I wouldn't mind waiting these 15 seconds on the client side to display this amount of data because that does not affect running BSB-LAN. The problem is rather waiting for the server to send (in your case 14 MB of) data. I just ran a check on a 350kB file and the performance is around 150-180 kB/s. Weirdly, the Due isn't much slower than the ESP32 Olimex EVB, and even more weirdly, performance is significantly lower (approx. 50-80kB/s) when using WiFi compared to Ethernet (not sure why, the WiFi connection should not be a problem, BSB-LAN sits right next to the router).

So to transfer a 14 MB file would take 1.5 minutes to download the data on the Due and (Ethernet) Olimex during which BSB-LAN couldn't do anything else, i.e. queries or logging entries during that time would not be possible. On the ESP32, there might be a chance to handle the data preparation and sending in a different core, but not on the Due.
The advantage of this approach would be that moving through the data could be done without additional data transfers. The drawback is that renewing the data would always fetch the complete file again, taking another 1.5 minutes.

I guess I'll have to run some performance tests with the ESP32 SQLite implementation and see if that significantly speeds things up on the server side. If it doesn't then I'd happily take you up on your offer.

@fredlcore
Copy link
Owner

I just ran the tests that come with the ESP32 SQLite implementation:
Counting all 9000 rows in a randomly generated database takes around 3.5 seconds.
Running a query across these 9000 entries that match a certain condition and list the first 10 of them takes 0.25 seconds - including the time it takes to output the contents on the serial console.

@DE-cr
Copy link
Contributor

DE-cr commented Feb 20, 2023 •
edited
Loading

An easy solution for people logging much data and usually just wanting to display the newest entries would be to add a URL command to output the newest n kB of /D only, e.g. /DEn or maybe just /DE with n fixed or configurable in /C (/DE = data end). This could very easily be served by seeking n kB back from the end of the open datalog.txt, skipping to the first \n from there, and delivering the rest. If the user should miss the header line in this approach, that could also be easily fixed (but the current /DG implementation doesn't need it: it just skips the first line).

This could be implemented in addition to my current /DG modification, which could then use e.g. /DE100 per default to fetch the data, with a possible extension to fetch /D completely on request.

And yes, it seems C3 and/or D3 has not been designed to handle large datasets. However, I'm still amazed by what it does, and how easy it is to use - for free.

@DE-cr
Copy link
Contributor

DE-cr commented Mar 16, 2023

One very important aspect in my decision to not do /DKn in-place that I forgot to mention:
If any file I/O goes wrong, the old files would already be unusable. With the currently implemented approach, that's only true after the actual data copying is completed, when replacing the old files with the new files (remove and rename).

@DE-cr
Copy link
Contributor

DE-cr commented Mar 17, 2023

One very important aspect in my decision to not do /DKn in-place that I forgot to mention: If any file I/O goes wrong, the old files would already be unusable. With the currently implemented approach, that's only true after the actual data copying is completed, when replacing the old files with the new files (remove and rename).

I think it was the right decision => #557 (comment) :)

@DE-cr
Copy link
Contributor

DE-cr commented Mar 18, 2023

Fyi, @CZvacko:
The final (? ;) version to address this feature request has now been merged into the bsb-lan master.
If you're willing to let your system consume about 1.7 kWh more per year, I recommend to also disable setCpuFrequencyMhz(80); in BSB_LAN.ino for improved speed in /DG, /D, etc.

@CZvacko
Copy link
Contributor Author

CZvacko commented Mar 19, 2023

@DE-cr just installed it and I'm going to try it. But I had to execute D0, so I have to wait for the data...

But I noticed some problem, I happened also with your version earlier:
In the main menu, when I click on "Sensors", webbrowser opens sensors sub-page (K43) but only 3 menu items are displayed and the rest of the page content is missing. sensors
I personally don't use such a feature, but can this possibly happen to anyone else ? In my BSB_LAN_config.h I set the byte mqtt_mode = 0; could this be the reason ?

@DE-cr
Copy link
Contributor

DE-cr commented Mar 19, 2023

http://bsb-lan/K43 works for me (I think):
grafik

@fredlcore
Copy link
Owner

The sensor category is at a different number each time because it depends on the total number of categories provided by the heater. If you query a non-existent category (such as maybe /K43 in your example), it will result undefined behaviour (either category 0 or all categories). It should look like this:

Bildschirm­foto 2023-03-19 um 09 37 19

But this has nothing to do with this Feature Request, so please open up a new issue and follow the bug reporting instructions provided.

@fredlcore
Copy link
Owner

I think with the new functions added by @DE-cr, we can close this issue here. Thanks a lot for this great work, this really makes the graph function of BSB-LAN much more usable, especially for stand-alone users!

@fredlcore
Copy link
Owner

Just when I closed this, I realize that there are gaps in my graph which seem to be what you described earlier. I thought that you had a solution for this? Or is it possible to configure the graph enginge to connect the lines even if there is no data inbetween?
Bildschirm­foto 2023-03-21 um 14 24 39

@DE-cr
Copy link
Contributor

DE-cr commented Mar 21, 2023

Could you send me the datalog (excerpt) that shows these gaps?

@fredlcore
Copy link
Owner

fredlcore commented Mar 21, 2023 via email •
edited
Loading

@fredlcore
Copy link
Owner

fredlcore commented Mar 21, 2023 via email •
edited
Loading

@DE-cr
Copy link
Contributor

DE-cr commented Mar 21, 2023

The problem here is that getting the value for parameter 8326 frequently takes longer than I had anticipated:
grafik
The /DG JavaScript code looks at the ms values to decide if a new log interval has started: If it differs by more than 999 from the previous ms value, /DG decides to start a new parameter set. Here, this leads to incomplete data sets to both sides of parameter 8326.

The fix would be straightforward: adjust the 999 in the JavaScript code (and the 1000 in the preceding comment) to a higher value. The question then is: How high do you want this value to set? My choice would be not more than 1900, to at least allow real log intervals of 2 s. Parameters with ms values less than that value apart would be allotted to the same "log interval" in the plots, i.e. with a real log interval of 1 s, you would see all parameter values placed at a single point of time in /DG!

Do you want me to prepare a PR? With the value set to 1900?

@DE-cr
Copy link
Contributor

DE-cr commented Mar 21, 2023

Btw, I currently log parameters where gaps are intended - because they frequently contain non-numerical values ("---"):
grafik
That straight long line spans a period when I didn't have logging enabled. For these events, there is no gap in the plot.

@fredlcore
Copy link
Owner

The problem here is that getting the value for parameter 8326 frequently takes longer than I had anticipated:

Yes, that is a problem with heating systems using an OCI420 LPB clip-in module which is terribly slow. It is used in LMU7 controllers when you connect them via LPB to a master controller such as an ISR SSR. Querys of parameters take up to 2 seconds, plus the overhead for switching from the standard device to the heater. These heaters have been sold a log around 10 years ago and if you want to upgrade them with an auxiliary heat source (solar, pellets etc.) you end up with such a configuration that I have now. If increasing the abovementioned time will help, I think it would be best to increase it to 3 seconds. I don't see where a log interval of 2 seconds would be reasonable, plus, there is a minimum log interval of 10 seconds currently in the code. So that wouldn't work anyways. Or did you mean something else?

@DE-cr
Copy link
Contributor

DE-cr commented Mar 21, 2023

If bsb-lan reliably limits the log interval to >=10s, I would change 999 to 9999 in the code.
Shall I create a PR for that?

@DE-cr
Copy link
Contributor

DE-cr commented Mar 21, 2023

I'll just do one, and let you decide to either merge it or reject it. :)

@DE-cr DE-cr mentioned this issue Mar 21, 2023
Merged
@fredlcore
Copy link
Owner

Hm, the line is currently commented out (6588 in BSB_LAN.ino) and would only apply to the URL-command /L, but I will see to activate it at the proper place because even 10 seconds is already blocking BSB-LAN most of the time (with my 6-8 parameters every 10 seconds, it was someteims hard to access the webinterface because BSB-LAN is constantly querying the parameters).

@DE-cr
Copy link
Contributor

DE-cr commented Mar 22, 2023 •
edited
Loading

Btw, I've now chosen 10s as a value to recognize the start of a new data set for the following reasons:

  • You've said that 10s would be the shortest log interval allowed/supported by bsb-lan. If shorter values are / should be supported, the current 10s value would have to be reduced.
  • In your data there was always a gap of >30s between the last data sample from a data set and the first sample of the next set, not only between data sets, i.e. the actual log interval used by bsb-lan seems to be the value set by the user plus the time it takes to sample a data set. If this behavior is changed, the 10s value should also be reduced to a value just large enough to cover the longest time it could take for bsb-lan to deliver a single data point. With all the data I have seen so far, I would set it to 1.5s.
  • Your suggested 3s looked too arbitrary to me.

@fredlcore
Copy link
Owner

Yes, it is the value set by the user plus the time it takes to sample a data set because quite a few users poll 30 parameters or more (40 is the software limit), and if they set a poll interval of 20 seconds or so, the next poll interval will be reached before the previous poll is complete, resulting in BSB-LAN immediately starting the next poll and thus constantly hogging the bus. With the current approach, it is made sure that there will at least be a timeframe of X seconds where there is no logging activity.

While 1.5 seconds may be fine with the current data set, you can never be certain that there is an event where BSB-LAN has to wait due to other devices bus activities which can then prolong the polling of a single data point even further. That's why 3s seemed a good compromise to me (a 100% safety margin). But since I can't think of a (reasonable) situation where an interval of less than 10s would make any sense, we can keep it as it is.

@DE-cr
Copy link
Contributor

DE-cr commented Mar 29, 2023

@DE-cr just installed it and I'm going to try it. But I had to execute D0, so I have to wait for the data...

But I noticed some problem, I happened also with your version earlier: In the main menu, when I click on "Sensors", webbrowser opens sensors sub-page (K43) but only 3 menu items are displayed and the rest of the page content is missing. sensors I personally don't use such a feature, but can this possibly happen to anyone else ? In my BSB_LAN_config.h I set the byte mqtt_mode = 0; could this be the reason ?

That was an issue unrelated to my changes for this feature request here, and has been fixed now: #564

@fredlcore
Copy link
Owner

@DE-cr: I got the recommendation to add this last line into the WiFi setup process (currently around line 7960) to limit the WiFi bandwidth to 20MHz instead of 40 MHz which should make WiFi connections more stable. Could you please test if this affects download transfer times (significantly)? Thanks!

  #ifdef ESP32
  WiFi.disconnect(true);  //disconnect form wifi to set new wifi connection
  WiFi.mode(WIFI_STA); //init wifi mode
  // Workaround for problems connecting to wireless network on some ESP32, see here: https://github.com/espressif/arduino-esp32/issues/2501#issuecomment-731618196
esp_wifi_set_bandwidth(WIFI_IF_STA, WIFI_BW_HT20);  // W.Bra. 23.03.23 HT20

Do not activate LR, this will more than significantly impact transfer speed and should only be used by users who do not use the graph display and have a long distance to the router.

@DE-cr
Copy link
Contributor

DE-cr commented Apr 1, 2023

I'd say it does not impact transfer speed in my setup (esp32 w/o sd card, wifi connection, ...).
After 50 repetitions of fetching 447 KB of /D data with either setting, I get almost the same numbers:
91/454/315 KB/s min/max/avg w/o that extra code line vs. 95/452/329.5 with the modification.

Upon startup, my bsb-lan unit in these tests had reported a signal strength of -80 vs. -78 (w/ mod).

Btw, I had to #include <esp_wifi.h>

@fredlcore
Copy link
Owner

Thanks for testing and the hint to the missing library. Added to master now.

@fredlcore
Copy link
Owner

@DE-cr: Not sure if this is an expected behaviour or an anomaly, but after I had to reset to disconnect BSB-LAN from the heater and restart it, BSB-LAN didn't seem to have been able to acquire the time and date from the heater. In this log, you can see the restart after the first few lines (millis() is reset) and then after around 16 minutes, it seems to have received the time from the heater and logging data is now correct. This has, however, the effect that log graph display is limited to that point where the date is no longer in the 1970s. Is there something that can be done about this (maybe based on the correctly running millis) or do we have to accept this? Both is fine with me, I just want to be able to explain this to users if such questions arise.

@DE-cr
Copy link
Contributor

DE-cr commented Apr 16, 2023

Could you provide me the output of /D and /DI to better look into this?

@fredlcore
Copy link
Owner

Sorry, forgot to paste:

91636388;16.04.2023 13:38:24;8310;Kesseltemperatur;40.0;°C
91637127;16.04.2023 13:38:24;8311;Kesselsollwert;37.5;°C
91637986;16.04.2023 13:38:25;8326;Brennermodulation;0;%
91638387;16.04.2023 13:38:26;8952;Schienenrücklauftemperatur;29.3;°C
91638581;16.04.2023 13:38:26;8980;Pufferspeichertemperatur 1;33.5;°C
91638771;16.04.2023 13:38:26;8982;Pufferspeichertemperatur 2;22.6;°C
281994;01.01.1970 00:04:42;8310;Kesseltemperatur;49.0;°C
283074;01.01.1970 00:04:43;8311;Kesselsollwert;16.0;°C
283793;01.01.1970 00:04:43;8326;Brennermodulation;0;%
284294;01.01.1970 00:04:44;8952;Schienenrücklauftemperatur;35.2;°C
284597;01.01.1970 00:04:44;8980;Pufferspeichertemperatur 1;33.2;°C
284897;01.01.1970 00:04:44;8982;Pufferspeichertemperatur 2;22.5;°C
315575;01.01.1970 00:05:15;8310;Kesseltemperatur;46.5;°C
316806;01.01.1970 00:05:16;8311;Kesselsollwert;16.0;°C
317891;01.01.1970 00:05:17;8326;Brennermodulation;0;%
318382;01.01.1970 00:05:18;8952;Schienenrücklauftemperatur;33.4;°C
318701;01.01.1970 00:05:18;8980;Pufferspeichertemperatur 1;33.2;°C
319001;01.01.1970 00:05:19;8982;Pufferspeichertemperatur 2;22.5;°C
349696;01.01.1970 00:05:49;8310;Kesseltemperatur;45.0;°C
350835;01.01.1970 00:05:50;8311;Kesselsollwert;16.0;°C
352042;01.01.1970 00:05:52;8326;Brennermodulation;0;%
352539;01.01.1970 00:05:52;8952;Schienenrücklauftemperatur;32.3;°C
352845;01.01.1970 00:05:52;8980;Pufferspeichertemperatur 1;33.2;°C
353150;01.01.1970 00:05:53;8982;Pufferspeichertemperatur 2;22.5;°C
384064;01.01.1970 00:06:24;8310;Kesseltemperatur;44.0;°C
385303;01.01.1970 00:06:25;8311;Kesselsollwert;16.0;°C
386352;01.01.1970 00:06:26;8326;Brennermodulation;0;%
386861;01.01.1970 00:06:26;8952;Schienenrücklauftemperatur;31.5;°C
387164;01.01.1970 00:06:27;8980;Pufferspeichertemperatur 1;33.2;°C
387467;01.01.1970 00:06:27;8982;Pufferspeichertemperatur 2;22.5;°C
418336;01.01.1970 00:06:58;8310;Kesseltemperatur;42.5;°C
419244;01.01.1970 00:06:59;8311;Kesselsollwert;15.5;°C
420483;01.01.1970 00:07:00;8326;Brennermodulation;0;%
420972;01.01.1970 00:07:01;8952;Schienenrücklauftemperatur;30.7;°C
421277;01.01.1970 00:07:01;8980;Pufferspeichertemperatur 1;33.2;°C
421567;01.01.1970 00:07:01;8982;Pufferspeichertemperatur 2;22.5;°C
452388;01.01.1970 00:07:32;8310;Kesseltemperatur;41.0;°C
453271;01.01.1970 00:07:33;8311;Kesselsollwert;15.5;°C
454590;01.01.1970 00:07:34;8326;Brennermodulation;0;%
455098;01.01.1970 00:07:35;8952;Schienenrücklauftemperatur;30.0;°C
455398;01.01.1970 00:07:35;8980;Pufferspeichertemperatur 1;33.2;°C
455693;01.01.1970 00:07:35;8982;Pufferspeichertemperatur 2;22.5;°C
486560;01.01.1970 00:08:06;8310;Kesseltemperatur;40.0;°C
487459;01.01.1970 00:08:07;8311;Kesselsollwert;15.5;°C
488672;01.01.1970 00:08:08;8326;Brennermodulation;0;%
489171;01.01.1970 00:08:09;8952;Schienenrücklauftemperatur;29.7;°C
489459;01.01.1970 00:08:09;8980;Pufferspeichertemperatur 1;33.2;°C
489753;01.01.1970 00:08:09;8982;Pufferspeichertemperatur 2;22.5;°C
520510;01.01.1970 00:08:40;8310;Kesseltemperatur;38.5;°C
521394;01.01.1970 00:08:41;8311;Kesselsollwert;15.5;°C
522664;01.01.1970 00:08:42;8326;Brennermodulation;0;%
523157;01.01.1970 00:08:43;8952;Schienenrücklauftemperatur;29.4;°C
523466;01.01.1970 00:08:43;8980;Pufferspeichertemperatur 1;33.2;°C
523762;01.01.1970 00:08:43;8982;Pufferspeichertemperatur 2;22.5;°C
554546;01.01.1970 00:09:14;8310;Kesseltemperatur;37.0;°C
555428;01.01.1970 00:09:15;8311;Kesselsollwert;17.0;°C
556173;01.01.1970 00:09:16;8326;Brennermodulation;0;%
556655;01.01.1970 00:09:16;8952;Schienenrücklauftemperatur;29.2;°C
556958;01.01.1970 00:09:17;8980;Pufferspeichertemperatur 1;33.2;°C
557244;01.01.1970 00:09:17;8982;Pufferspeichertemperatur 2;22.5;°C
588011;01.01.1970 00:09:48;8310;Kesseltemperatur;35.0;°C
589094;01.01.1970 00:09:49;8311;Kesselsollwert;17.0;°C
590329;01.01.1970 00:09:50;8326;Brennermodulation;0;%
590836;01.01.1970 00:09:50;8952;Schienenrücklauftemperatur;29.1;°C
591130;01.01.1970 00:09:51;8980;Pufferspeichertemperatur 1;33.2;°C
591432;01.01.1970 00:09:51;8982;Pufferspeichertemperatur 2;22.5;°C
622210;01.01.1970 00:10:22;8310;Kesseltemperatur;34.0;°C
623086;01.01.1970 00:10:23;8311;Kesselsollwert;17.0;°C
623976;01.01.1970 00:10:24;8326;Brennermodulation;0;%
624454;01.01.1970 00:10:24;8952;Schienenrücklauftemperatur;28.8;°C
624767;01.01.1970 00:10:24;8980;Pufferspeichertemperatur 1;33.2;°C
625062;01.01.1970 00:10:25;8982;Pufferspeichertemperatur 2;22.5;°C
655691;01.01.1970 00:10:55;8310;Kesseltemperatur;33.0;°C
656800;01.01.1970 00:10:56;8311;Kesselsollwert;17.0;°C
658045;01.01.1970 00:10:58;8326;Brennermodulation;0;%
658538;01.01.1970 00:10:58;8952;Schienenrücklauftemperatur;28.8;°C
658839;01.01.1970 00:10:58;8980;Pufferspeichertemperatur 1;33.2;°C
659142;01.01.1970 00:10:59;8982;Pufferspeichertemperatur 2;22.5;°C
690021;01.01.1970 00:11:30;8310;Kesseltemperatur;32.5;°C
690891;01.01.1970 00:11:30;8311;Kesselsollwert;20.0;°C
691955;01.01.1970 00:11:32;8326;Brennermodulation;0;%
692440;01.01.1970 00:11:32;8952;Schienenrücklauftemperatur;28.7;°C
692757;01.01.1970 00:11:32;8980;Pufferspeichertemperatur 1;33.2;°C
693063;01.01.1970 00:11:33;8982;Pufferspeichertemperatur 2;22.5;°C
723872;01.01.1970 00:12:03;8310;Kesseltemperatur;31.5;°C
724912;01.01.1970 00:12:05;8311;Kesselsollwert;24.5;°C
726001;01.01.1970 00:12:06;8326;Brennermodulation;0;%
726491;01.01.1970 00:12:06;8952;Schienenrücklauftemperatur;28.5;°C
726801;01.01.1970 00:12:06;8980;Pufferspeichertemperatur 1;33.2;°C
727095;01.01.1970 00:12:07;8982;Pufferspeichertemperatur 2;22.5;°C
758011;01.01.1970 00:12:38;8310;Kesseltemperatur;31.0;°C
759049;01.01.1970 00:12:39;8311;Kesselsollwert;25.5;°C
760131;01.01.1970 00:12:40;8326;Brennermodulation;0;%
760644;01.01.1970 00:12:40;8952;Schienenrücklauftemperatur;28.4;°C
760944;01.01.1970 00:12:41;8980;Pufferspeichertemperatur 1;33.2;°C
761246;01.01.1970 00:12:41;8982;Pufferspeichertemperatur 2;22.5;°C
791894;01.01.1970 00:13:11;8310;Kesseltemperatur;30.5;°C
793108;01.01.1970 00:13:13;8311;Kesselsollwert;29.0;°C
794180;01.01.1970 00:13:14;8326;Brennermodulation;0;%
794863;01.01.1970 00:13:14;8952;Schienenrücklauftemperatur;28.4;°C
795159;01.01.1970 00:13:15;8980;Pufferspeichertemperatur 1;33.2;°C
795455;01.01.1970 00:13:15;8982;Pufferspeichertemperatur 2;22.5;°C
826430;01.01.1970 00:13:46;8310;Kesseltemperatur;29.5;°C
827503;01.01.1970 00:13:47;8311;Kesselsollwert;30.5;°C
828585;01.01.1970 00:13:48;8326;Brennermodulation;0;%
829080;01.01.1970 00:13:49;8952;Schienenrücklauftemperatur;28.3;°C
829377;01.01.1970 00:13:49;8980;Pufferspeichertemperatur 1;33.2;°C
829680;01.01.1970 00:13:49;8982;Pufferspeichertemperatur 2;22.5;°C
860618;01.01.1970 00:14:20;8310;Kesseltemperatur;29.0;°C
861518;01.01.1970 00:14:21;8311;Kesselsollwert;33.5;°C
862739;01.01.1970 00:14:22;8326;Brennermodulation;13;%
863242;01.01.1970 00:14:23;8952;Schienenrücklauftemperatur;28.3;°C
863553;01.01.1970 00:14:23;8980;Pufferspeichertemperatur 1;33.2;°C
863849;01.01.1970 00:14:23;8982;Pufferspeichertemperatur 2;22.5;°C
894736;01.01.1970 00:14:54;8310;Kesseltemperatur;32.5;°C
895972;01.01.1970 00:14:56;8311;Kesselsollwert;35.5;°C
897073;01.01.1970 00:14:57;8326;Brennermodulation;2;%
897566;01.01.1970 00:14:57;8952;Schienenrücklauftemperatur;28.3;°C
897864;01.01.1970 00:14:57;8980;Pufferspeichertemperatur 1;33.2;°C
898162;01.01.1970 00:14:58;8982;Pufferspeichertemperatur 2;22.5;°C
928967;01.01.1970 00:15:29;8310;Kesseltemperatur;34.0;°C
930022;01.01.1970 00:15:30;8311;Kesselsollwert;35.5;°C
931324;01.01.1970 00:15:31;8326;Brennermodulation;4;%
931826;01.01.1970 00:15:31;8952;Schienenrücklauftemperatur;28.1;°C
932121;01.01.1970 00:15:32;8980;Pufferspeichertemperatur 1;33.2;°C
932415;01.01.1970 00:15:32;8982;Pufferspeichertemperatur 2;22.5;°C
963181;01.01.1970 00:16:03;8310;Kesseltemperatur;35.0;°C
964459;01.01.1970 00:16:04;8311;Kesselsollwert;35.5;°C
965550;01.01.1970 00:16:05;8326;Brennermodulation;0;%
966041;01.01.1970 00:16:06;8952;Schienenrücklauftemperatur;28.1;°C
966335;01.01.1970 00:16:06;8980;Pufferspeichertemperatur 1;33.2;°C
966632;01.01.1970 00:16:06;8982;Pufferspeichertemperatur 2;22.5;°C
997487;01.01.1970 00:16:37;8310;Kesseltemperatur;35.5;°C
998356;01.01.1970 00:16:38;8311;Kesselsollwert;36.0;°C
999231;01.01.1970 00:16:39;8326;Brennermodulation;1;%
999752;01.01.1970 00:16:39;8952;Schienenrücklauftemperatur;28.1;°C
1000066;01.01.1970 00:16:40;8980;Pufferspeichertemperatur 1;33.2;°C
1000362;01.01.1970 00:16:40;8982;Pufferspeichertemperatur 2;22.5;°C
1031079;01.01.1970 00:17:11;8310;Kesseltemperatur;36.0;°C
1032101;01.01.1970 00:17:12;8311;Kesselsollwert;37.5;°C
1033331;01.01.1970 00:17:13;8326;Brennermodulation;1;%
1033831;01.01.1970 00:17:13;8952;Schienenrücklauftemperatur;28.1;°C
1034136;01.01.1970 00:17:14;8980;Pufferspeichertemperatur 1;33.2;°C
1034438;01.01.1970 00:17:14;8982;Pufferspeichertemperatur 2;22.5;°C
1065366;01.01.1970 00:17:45;8310;Kesseltemperatur;36.5;°C
1066608;01.01.1970 00:17:46;8311;Kesselsollwert;37.5;°C
1067681;01.01.1970 00:17:47;8326;Brennermodulation;4;%
1068175;01.01.1970 00:17:48;8952;Schienenrücklauftemperatur;28.1;°C
1068475;01.01.1970 00:17:48;8980;Pufferspeichertemperatur 1;33.2;°C
1068781;01.01.1970 00:17:48;8982;Pufferspeichertemperatur 2;22.5;°C
1099646;16.04.2023 16:59:37;8310;Kesseltemperatur;36.5;°C
1100699;16.04.2023 16:59:38;8311;Kesselsollwert;37.5;°C
1101393;16.04.2023 16:59:39;8326;Brennermodulation;0;%
1101789;16.04.2023 16:59:39;8952;Schienenrücklauftemperatur;28.0;°C
1101977;16.04.2023 16:59:39;8980;Pufferspeichertemperatur 1;33.2;°C
1102170;16.04.2023 16:59:40;8982;Pufferspeichertemperatur 2;22.5;°C
1132899;16.04.2023 17:00:10;8310;Kesseltemperatur;37.0;°C
1133819;16.04.2023 17:00:11;8311;Kesselsollwert;39.5;°C

@fredlcore
Copy link
Owner

And /DI:

Date;DatalogPosition
2023-03-21;52
2023-03-22;674315
2023-03-23;2250559
2023-03-24;3941614
2023-03-25;5653306
2023-03-26;7356838
2023-03-27;8969980
2023-03-28;10174618
2024-03-28;11404846
2023-03-28;11407230
2023-03-29;11659576
2023-03-30;13184281
2023-03-31;14711512
2023-04-01;16246029
2023-04-02;17792888
2023-04-03;19310366
2023-04-04;20867503
2023-04-05;22435009
2023-04-06;24003556
2023-04-07;25572676
2023-04-08;27131768
2023-04-09;28666097
2023-04-10;30181599
2023-04-11;31705954
2023-04-12;32835591
2023-04-13;33844364
2023-04-14;34867061
2023-04-15;35891523
2023-04-16;36903700
1970-01-01;37477768
2023-04-16;37486879

@DE-cr
Copy link
Contributor

DE-cr commented Apr 16, 2023

Thanks.

What you're experiencing here with /DG is by design:

  • Upon reset, both the first and last entry shown in /DI are read from the datalog index file datalog.idx
  • They are then used as the min (first entry) and max (last entry) date selectable with the GUI date controls in /DG (because normally, dates shouldn't run backwards, and therefore the index file should contain dates in ascending order only).
  • When /DG is invoked, it tries to show data from the most recent DEFAULT_DAYS_TO_PLOT (as defined in BSB_LAN_config.h) as of (the browser's idea of) now.
  • It does this by reading the index file backwards (c.f. [FEATURE REQUEST] Modifying the graph to allow the long-term data logging #539 (comment) ff), stopping when it finds a date before the desired range - which 1970 clearly is, unless you define DEFAULT_DAYS_TO_PLOT to reach that far. ;)

Of course, different designs would be possible. Some of them have been discussed above, but this is what we've decided upon.

Personally, I wouldn't consider displaying data from the year 1970 together with today's data useful: Even on a UHD display, that would mean about five days sharing a single pixel in the plot. Also, your "1970" data would be displayed on the very left, not where it really belongs.

What I would do with such a datalog:

  • save it (/D)
  • then either /DK1 to erase everything before offset 37486879 from it (2023-04-16;37486879)
  • or remove (or fix, if you know how) the "1970" lines in the data downloaded, then

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fredlcore @1coderookie @CZvacko @DE-cr