New logging mechanics

[rafalsk]

Hey, so we've decided to migrate our code-base to recent version of Mongoose.

Looking good going fine (..) so far.

Thought I would share my thoughts about the seemingly refactored logging mechanics.

As of now, by default, at a certain moment of processing the library has a full string for each log message on the stack. Now, the custom callback function definition has changed. It now no longer is given the full buffer along with its size but is invoked on a per-byte basis through logc().

Here are my doubts/comments:

• I am not sure if say 256 calls to logc() are more efficient than a single memcpy
• the custom log function looses the ability to detect log message boundaries

May I dare to ask what was the rationale?

static void mylog(char ch, void *param) {
  static char buf[256];
  static size_t len;
  buf[len++] = ch;
  if (ch == '\n' || len >= sizeof(buf)) {
    syslog(LOG_INFO, "%.*s", (int) len, buf); // Send logs
    len = 0;
  }
}

I found the above sample handler in docs. Previously nobody prevented the callback function from doing in-ram buffering on a per-message basis.

also please excuse this off-the-topic question, but - why does custom-file-system handlers now return void* instead of a concretized pointer of a file descriptor?

[scaprile]

From what I remember (and IIRC): We got completely rid of any use of printf()-like functions. Some useful extensions required a heavy use of memory allocation and/or guessing abilities; these was greatly simplified by using character-based output, keeping the UI simple and so less error-prone.

My impression is that void * is used to be able to keep generality while coexisting with the myriad of architectural options where Mongoose runs or will run.

Hope this answers your questions

[rafalsk]

@scaprile thank you for your response.

void * might be much more general, but it is way more error-prone.

Notice that delete (void*) does not take into account the actual memory allocation of the object and the caller of delete needs to be aware of the object type, do reinterpret_cast to the concrete pointer and only then call delete.

Were there any changes made to how API for connections is used?

In our project (https://gridnet.org) we take extremely heavy use of Mongoose.

Were there any changes done to connection's API?

I mean, without doing any changes to our-code base, when we form connections to external endpoints, with Mongoose acting as client (..) with TLS enables (or even without I dunno) the socket does never transition into a 'writable' 'edible' state Even though the CONNECTED callback had fired.. We never receive any response from a remote endpoint. We haven't investigated this further though we've been spending an entire day porting.. without much success so far. Calling same functions and stuff....

From the API perspective, the logging callback changed (..) the file-system handlers changed but still we are unable to get basic data-connectivity up and running with Mongoose acting as a client.

[rafalsk]

We are porting from version 7.4

Were there changes made to how the user-defined-object is passed along the callbacks?
Without doing any changes to our code-base we're seeing bad memory allocations pointed to by 'fn_data'.

[scaprile]

Lots of changes. I'm familiar with 7.7+, so let's wait for @cpq himself.

In the meantime, I suggest you take a look at the tutorials, they are up to date. http://mongoose.ws/tutorials

What is your OS/hardware ?

[rafalsk]

Lots of changes. I'm familiar with 7.7+, so let's wait for @cpq himself.

ok then, let us wait..

as you may see below, our production version based on 7.4 works 'juuust fine':

https://test.gridnet.org:444/https://google.com
https://test.gridnet.org:444/https://wikipedia.org

Decentralized UI:

https://test.gridnet.org

It's Mongoose everywhere.

we've also noticed the object representing remote peer from connection got ridden of, it's all good all fine.

[scaprile]

Great! (Note I just edited the previous answer)

[rafalsk]

Great! (Note I just edited the previous answer)

we're running on MS Windows 11/Server 2019/2022 64 bit compilation

[rafalsk]

In the meantime, I suggest you take a look at the tutorials, they are up to date. http://mongoose.ws/tutorials

Not being lazy; I was just hoping for some hints on what to keep in mind / look-our for, especially taking into account that we've done no changes to our code base and the API stayed the same, for the most of it. We are working with RAW data received from sockets i.e. we rely on MG_EV_READ, we pass our custom object around in fn_data, for each connection.

In your example of a handler you've got static size_t len; Do notice that it is not guaranteed to be initialized with 0 in a Release build, with optimizations enabled.

[rafalsk]

Hello? Are these platform specific issues we're experiencing?

[scaprile]

I don't think so, but if you send us a minimal example of what doesn't work I might check.

[cpq]

@rafalsk

Mongoose uses its own family of printf functions, where the lowest level is this:

size_t mg_xprintf(void (*out)(char, void *), void *param, const char *fmt, ...);

It outputs a formatted string to a "printer function". We chose a signature for that "printer function" to be a per-byte printer:

typedef void (*mg_pfn_t)(char, void *);

The alternative was to use a "memory chunk" printer:

typedef void (*mg_pfn_t)(const void *buf, size_t len, void *);

Why did we use a per-byte printer? Because it is simpler. Yes it is not that as efficient as an alternative, but in our design priorities, simplicity stays above performance.

In this particular case, the alternative is also simple enough, so we might reconsider - if there is an evidence that the performance hit is significant in a real production use cases.

[rafalsk]

@cpq @scaprile kindly thank you for your assistance and all of your responses.

Since, as of now, the conversations shed no light on the possible reasons behind the issues we've seemed to encounter with versions > 7.4,
I shall leave this to our in-house research and report once I have more information available.

For now we'll keep riding 7.4 which simply works for us without any issue at all - can't thus afford more hours on this, as of now.

[cpq]

@rafalsk thanks.

Though re-reading what you've written, I still don't understand what's your issue.
If you happen to have a something concrete , please file an issue.

[rafalsk]

yet again thank you for your reply; the issue was not specified since, like I said - the story was simple - we've switched to latest version from 7.4. We've upgraded codebase taking into account some if the imminent API changes.

And then, we began seeing lots of underlying errors, objects pointed to by fn_data not being set properly etc.

Which as you presumably would agree would either require us to spend lots of time investigating the issues (whcih were sorta unexpected since 7.4 works perfect for us ) or, I thought I would get right over here and ask whether there where any significant changes made to the API (..) etc. user objects pointed by fn_data needed to be registered somewhere, whatever,

Like I said, once we have some time resources to try again we would lay out the issues we've faced in more detail.

For now, I've had 3 developers looking into it for entire day and we've decided to switch back to 7.4 since we couldn't really tackle what was going on within the expected timeframe.

Thanks for the lib.

[cpq]

@rafalsk please refrain from wasting yours and our time on vague talks