ARTEMIS-4206 Dealing with Large Message Issues / duplicate deliveries / messages hanging on folder

[clebertsuconic]

you add the property -DARTEMIS_DEBUG_REF=true in your java arguments, And you should see Debug output when the messages are not being removed.

[clebertsuconic]

I have been debugging with these messages, and I am making a few adjustments (Missed debug statements... etc)... this is being useful in a scenario where the Cpu Is overloaded and everything is failing on the delivering, leaving messages behind.

it is working as I expected to tell me what is happening, but I'm making a few adjustments.

[gemmellr]

I know its still draft but as its been a couple days and you keep updating it..some WIP comments.

[gemmellr]

You were right, I didnt like that there were two sets of changes in here, intertwined no less :)

I'm not a huge fan of using the log levels to decide to randomly start changing behaviour and stashing stacktraces, even if the intent is sometimes log them...or of using that mechanism throughout the codebase rather than just inside the message classes as it was originally.

[gemmellr]

All the others have timestamps, seems weird this doesnt.

[clebertsuconic]

I can squash 2 commits in one for two different and related issues. I thought it was better to keep separated.

Regarding the trace. I think it's better this way. Say an user starts having issues if we just set the logger we would get what we need.

[clebertsuconic]

This mechanism is how I found these issues. I would like to keep this door open for future issues.

[clebertsuconic]

The idea of this feature is to be a hidden feature...

So, I am going to use the following logger to enable this type of debugging:

private static final Logger refLogger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass().getName() + ".REF_DEBUG");

[gemmellr]

I can squash 2 commits in one for two different and related issues. I thought it was better to keep separated.

No, lets not squash 'different changes' into one commit (yes I saw, its too late), lets keep different changes separate. In that spirit, the original comment was around lets also try not to make different changes entirely reliant on each other anyway by intertwining and overlapping their multiple commits, such that you cant very simply pick or see the effect of one of the different changes in isolation because the commits were mixed together rather than sequential while adding.

Aside, on the squashed commit log message, it starts telling you how to enable the ref counting debug, and then also ends telling you how to enable the ref counting debug but in a less clear way. Seems like one of them could go.

[gemmellr]

The idea of this feature is to be a hidden feature...

So, I am going to use the following logger to enable this type of debugging:

private static final Logger refLogger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass().getName() + ".REF_DEBUG");

It will certainly be hidden, people will enable it without knowing they are since it is a descendent in the regular class name logger hierarchy. Anyone simply enabling debug logging for the class (or more likely, many classes / further up the hierarchy) will get the change in actual system behaviour of the ref counting debug and saving lots of stacktraces. They would have to know about the change in behaviour and reconfigure the specific ref count logger back off in order to restore the prior behaviour and only add debug output for the normal system behaviour.

[michaelandrepearce]

The idea of this feature is to be a hidden feature...

Hmmm this worries me, I think any feature should be something we're open about have the capability documented etc. We shouldn't be hiding things so only select few know about it.

This is ethos of open source Apache in my mind.

Fair enough if it's something a general user may not need to know, call it like advanced troubleshooting or something but do not add something hidden

[gemmellr]

The idea of this feature is to be a hidden feature...

Hmmm this worries me, I think any feature should be something we're open about have the capability documented etc. We shouldn't be hiding things so only select few know about it.

This is ethos of open source Apache in my mind.

Fair enough if it's something a general user may not need to know, call it like advanced troubleshooting or something but do not add something hidden

Yeah, its not a 'feature-feature' in any way, and not something I think a user would normally care about so I dont think I'd ever document it personally. Its 'additional broker developer leak debug/trace details via saved stacktraces'.

[clebertsuconic]

@gemmellr I will keep the squash.. they are all related anyways.

[clebertsuconic]

@michaelandrepearce I didn't mean as an actual hidden property. I will have a JIRA open for this that will be part of the release notes.

Just that I won't add it to the documentation... as most users won't care about it.. and hopefully won't need to.

Over the years I have been uncommenting code when I needed to debug things... I think it's time to make that thing as part of the codebase now.