Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use C++ <random> instead of Qt or POSIX #460

Merged
merged 13 commits into from Apr 28, 2022
Merged

Conversation

ulmus-scott
Copy link
Contributor

This also removes the conditional compilation due to the Qt random number generator functions added in 5.10.

Checklist

@kmdewaal
Copy link
Contributor

This pull request does not only change the random function but it has also changes in the range of the random numbers in some places.

@garybuhrmaster
Copy link
Contributor

This also removes the conditional compilation due to the Qt random number generator functions added in 5.10

Another alternative, which would appear more consistent with the MythTV coding standards documentation, would be that after the (imminent) release of v32 to advocate for bumping the minimum QT level(*) and cleaning up the ifdef'ery if the core devs are too annoyed with such.

But that would be something for master-next (v33-Pre) after checking what the various target OS's now have as their shipping version (I suspect 5.12 could be the new minimum, but I have not checked all the various target platforms).

(*) As I recall, with the current master, the reasoning for Qt 5.9 as a minimum was that at the time a few of the targeted OS's were at 5.9.  I think all of those OS's have either aged out of the targeted platform list, or are otherwise no longer supported for master (due to other requirements).

@ulmus-scott
Copy link
Contributor Author

https://pkgs.org/search/?q=Qtbase
It appears CentOS 7 and Ubuntu 18.04 LTS are still Qt 5.9.

Ubuntu 20.04 LTS is 5.12. Ubuntu 22.04 LTS should be 5.15 like everything else.

Of note, Qt 5.12 is EOL as of December.

@stuarta
Copy link
Member

stuarta commented Jan 13, 2022

https://pkgs.org/search/?q=Qtbase It appears CentOS 7 and Ubuntu 18.04 LTS are still Qt 5.9.

Ubuntu 20.04 LTS is 5.12. Ubuntu 22.04 LTS should be 5.15 like everything else.

Of note, Qt 5.12 is EOL as of December.

Centos 7 has already been dropped from master.
We won't explicitly support Ubuntu-18.04 LTS once 22.04 LTS is out.

@ulmus-scott
Copy link
Contributor Author

The POSIX random() function has now been replaced with MythRandom.

This was referenced Jan 20, 2022
@ulmus-scott
Copy link
Contributor Author

PacketBuffer::FreePacket no longer has any (reasonable) risk of errors from overflowing.

It was only used for a private variable, so hide it in the constructor in the .cpp file.

This drastically reduces the recompilations needed when changing mythrandom.h.
This forces the use of the unsigned overload, which technically may be a change.

However, we never actually request any negative numbers and most calls are of the form (0, x),
so this is functionally equivalent.
The file uses mythrandom.h instead.

Add include <cstdint>, which is also transitively included by "mythrandom.h".
m_next_empty_packet_key is a random number that is incremented.  In FreePacket(),
the upper 32 bits are tested for equivalence.

However, the incrementing of m_next_empty_packet_key could overflow the lower 32 bits,
changing the upper 32.  This would cause the if statement to be false and the packet to
not be marked as empty.

Starting with m_next_empty_packet_key with the lower 32 bits cleared, prevents this error
by allowing 2^32 packets before overflowing, far more than can be reasonably expected.
@ulmus-scott ulmus-scott changed the title Use C++ <random> instead of Qt Use C++ <random> instead of Qt or POSIX Mar 25, 2022
@linuxdude42
Copy link
Contributor

These changes look fine to me. I keep going back to look at the packetbuffer.cpp change because it does produce a different result than the original, but if the comment in packetbuffer.h is correct then this appears to be a better solution. The old solution filled the bottom 56 bits of m_next_empty_packet_key with a random number, where an increment (potentially the very first one) could overflow from the bottom 32 bits into the top 32 bits. This would cause PacketBuffer::FreePacket to stop caching packets with the old top 32 bits and only cache packets with the new top 32 bits. (Not that this would cause a problem, it would just churn packets for a while.) The new solution delays this churn basically indefinitely.

It does seems to me though that PacketBuffer::FreePacket never actually frees anything, and thus is a memory leak. Shouldn't packets that aren't added to m_empty_packets be deleted?

@ulmus-scott
Copy link
Contributor Author

These changes look fine to me. I keep going back to look at the packetbuffer.cpp change because it does produce a different result than the original, but if the comment in packetbuffer.h is correct then this appears to be a better solution. The old solution filled the bottom 56 bits of m_next_empty_packet_key with a random number, where an increment (potentially the very first one) could overflow from the bottom 32 bits into the top 32 bits.

I think the expected value of the initial packet buffer size would be 2^31 on average before overflowing and increasing in size to 2^32 after overflowing.

This would cause PacketBuffer::FreePacket to stop caching packets with the old top 32 bits and only cache packets with the new top 32 bits. (Not that this would cause a problem, it would just churn packets for a while.) The new solution delays this churn basically indefinitely.

On average it never happened before either (2^31).

It does seems to me though that PacketBuffer::FreePacket never actually frees anything, and thus is a memory leak. Shouldn't packets that aren't added to m_empty_packets be deleted?

If you look at udppacket.h and

/** \brief Frees an RTPDataPacket returned by PopDataPacket.
* \note Since we return UDPPackets and not pointers to packets
* we don't leak if this isn't called, but by preventing
* the reference count on the QByteArrays from going to
* 0 we can reuse the byte array and avoid mallocs.
*/
void FreePacket(const UDPPacket &packet);

UDPPacket is a reference counted QByteArray, so the memory would be freed when all references have fallen out of scope. It is not quite a memory leak because the packets can still be used in the future instead of creating new ones. If the UDPPacket were a fixed size, I think it would make more sense. A fixed size ring buffer might be a better choice instead of this.

Although, once the PacketBuffer goes out of scope, all of the memory should be freed.

So it is a change, but I don't think it's a significant one. We could also leave that for later and just use

MythRandom64() & ((UINT64_C(1) << 56) - 1)

.

@ulmus-scott
Copy link
Contributor Author

@linuxdude42 Some additional probabilities for overflowing:
The probability the old code overflowed given 2^N unprocessed packets was P = 2^(-32 + N); e.g. N = 10 (1024 packets), P = 1 / 4 194 304 .

@linuxdude42 linuxdude42 merged commit ca4d1bc into MythTV:master Apr 28, 2022
@ulmus-scott ulmus-scott deleted the random branch May 2, 2022 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants