Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve sending of large video frames in toxav. #718

Merged
merged 1 commit into from Feb 12, 2018

Conversation

iphydf
Copy link
Member

@iphydf iphydf commented Jan 21, 2018

This change does not include the addition of VP9. We do that in a
separate pull request.


This change is Reviewable

@iphydf iphydf added this to the v0.2.0 milestone Jan 21, 2018
@CLAassistant
Copy link

CLAassistant commented Jan 21, 2018

CLA assistant check
All committers have signed the CLA.

@jhert0
Copy link
Member

jhert0 commented Jan 21, 2018

Reviewed 8 of 8 files at r1.
Review status: all files reviewed at latest revision, 13 unresolved discussions.


toxav/rtp.c, line 153 at r1 (raw file):

        // if length(31 bits) > 1FFFFFFF then use all bits for length
        // and assume its a keyframe (most likely is anyway)
        if (LOWER_31_BITS(length_v3) > 0x1FFFFFFF) {

Create a constant for 0x1FFFFFFF


toxav/rtp.c, line 263 at r1 (raw file):

{
    assert(allocate_len >= data_length);
    struct RTPMessage *msg = (struct RTPMessage *)calloc(sizeof(struct RTPMessage) +

Make sure memory is allocated.


toxav/rtp.c, line 281 at r1 (raw file):

        uint32_t offset, uint32_t full_data_length, uint8_t is_keyframe)
{
    struct RTPMessage *msg = (struct RTPMessage *)calloc(1, sizeof(struct RTPMessage) + allocate_len);

Make sure memory is allocated.


toxav/rtp.c, line 303 at r1 (raw file):

static void move_slot(struct RTPWorkBufferList *wkbl, int8_t dst_slot, int8_t src_slot)
{
    memcpy(&(wkbl->work_buffer[dst_slot]), &(wkbl->work_buffer[src_slot]), sizeof(struct RTPWorkBuffer));

Add a check to make sure dst_slot isn't larger than USED_RTP_WORKBUFFER_COUNT


toxav/rtp.c, line 443 at r1 (raw file):

    if (offset_v3 > length_v3) {
        LOGGER_ERROR(log, "memory size too small!");

Shouldn't there be a return here or at least changing the offset?


toxav/rtp.c, line 496 at r1 (raw file):

    if (session_v3->work_buffer_list == NULL) {
        session_v3->work_buffer_list = (struct RTPWorkBufferList *)calloc(1, sizeof(struct RTPWorkBufferList));

Make sure memory is allocated.


toxav/rtp.c, line 541 at r1 (raw file):

        }

        if (slot == -1) {

Use else if


toxav/rtp.c, line 560 at r1 (raw file):

            if (slot == -2) {
                free(m_new);

Possible double free or freeing a NULL pointer.


toxav/rtp.c, line 602 at r1 (raw file):

        }

        if (slot == -1) {

Use else if


toxav/rtp.c, line 621 at r1 (raw file):

            if (slot == -2) {
                free(m_new);

Possible double free or freeing a NULL pointer.


toxav/rtp.c, line 627 at r1 (raw file):

        }

        if (slot > -1) {

Use else if


toxav/video.c, line 371 at r1 (raw file):

            (uint8_t)header_v3->pt == (rtp_TypeVideo % 128)) {
        LOGGER_DEBUG(vc->log, "rb_write msg->len=%d b0=%d b1=%d", (int)msg->len, (int)msg->data[0], (int)msg->data[1]);
        free(rb_write((RingBuffer *)vc->vbuf_raw, msg));

rb_write can return NULL.


toxav/video.c, line 373 at r1 (raw file):

        free(rb_write((RingBuffer *)vc->vbuf_raw, msg));
    } else {
        free(rb_write((RingBuffer *)vc->vbuf_raw, msg));

rb_write can return NULL.


Comments from Reviewable

@zoff99
Copy link

zoff99 commented Jan 21, 2018

changes:

  • fix the video bug (video frames larger than 65KBytes) by sending full frame length in alternate header field
  • improve video frame reconstruction logic with slots
  • configure video encoder and decoder to be multihtreaded
  • set error resilience flags on video codec
  • change encoder and decoder softdeadline

@iphydf
Copy link
Member Author

iphydf commented Jan 21, 2018

Review status: 7 of 8 files reviewed at latest revision, 13 unresolved discussions, some commit checks failed.


toxav/rtp.c, line 153 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Create a constant for 0x1FFFFFFF

@zoff99 what should we call it?


toxav/rtp.c, line 263 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Make sure memory is allocated.

Done.


toxav/rtp.c, line 281 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Make sure memory is allocated.

Done.


toxav/rtp.c, line 303 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Add a check to make sure dst_slot isn't larger than USED_RTP_WORKBUFFER_COUNT

Done.


toxav/rtp.c, line 443 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Shouldn't there be a return here or at least changing the offset?

@zoff99 what should we do when this happens?


toxav/rtp.c, line 496 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Make sure memory is allocated.

Done.


toxav/rtp.c, line 541 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Use else if

No need, there is a return in the previous one.


toxav/rtp.c, line 560 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Possible double free or freeing a NULL pointer.

free(NULL) is fine. How does a double free happen?


toxav/rtp.c, line 602 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Use else if

See above.


toxav/rtp.c, line 621 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Possible double free or freeing a NULL pointer.

See above.


toxav/rtp.c, line 627 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

Use else if

That would change the semantics.


toxav/video.c, line 371 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

rb_write can return NULL.

free(NULL) is safe.


toxav/video.c, line 373 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

rb_write can return NULL.

free(NULL) is safe.


Comments from Reviewable

@iphydf iphydf force-pushed the video-fix branch 2 times, most recently from 22114da to 3d5e37c Compare January 21, 2018 21:28
@jhert0
Copy link
Member

jhert0 commented Jan 21, 2018

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 4 unresolved discussions, some commit checks failed.


toxav/rtp.c, line 560 at r1 (raw file):

Previously, iphydf wrote…

free(NULL) is fine. How does a double free happen?

I'm not sure how likely it is and maybe its not but if m_new is not NULL and session->mcb is m_new is freed. Then slot is updated and if its -2 m_new is freed.


toxav/rtp.c, line 621 at r1 (raw file):

Previously, iphydf wrote…

See above.

same as above


Comments from Reviewable

@iphydf iphydf force-pushed the video-fix branch 3 times, most recently from 7f09b83 to b757955 Compare January 24, 2018 11:29
@iphydf
Copy link
Member Author

iphydf commented Jan 24, 2018

Review status: 7 of 8 files reviewed at latest revision, 4 unresolved discussions.


toxav/rtp.c, line 560 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

I'm not sure how likely it is and maybe its not but if m_new is not NULL and session->mcb is m_new is freed. Then slot is updated and if its -2 m_new is freed.

That doesn't seem correct. But more interestingly, m_new should always be null here:

  • If m_new is null and slot is -2, it's free(null).
  • If m_new is not null, then regardless of session->mcb, it is set to null, so this is free(null).

It's always free(null). @zoff99 can you look into this?


Comments from Reviewable

@iphydf
Copy link
Member Author

iphydf commented Jan 24, 2018

Review status: 7 of 8 files reviewed at latest revision, 4 unresolved discussions.


toxav/rtp.c, line 621 at r1 (raw file):

Previously, endoffile78 (Endoffile) wrote…

same as above

Same.


Comments from Reviewable

@iphydf iphydf changed the title Improve video key frame sending. WIP #2: Improve video key frame sending. Jan 25, 2018
@iphydf iphydf modified the milestones: v0.2.0, v0.2.x Jan 25, 2018
@iphydf iphydf force-pushed the video-fix branch 2 times, most recently from f884979 to 6d6bb96 Compare January 26, 2018 02:17
@sudden6
Copy link

sudden6 commented Jan 27, 2018

Reviewed 1 of 8 files at r1, 1 of 3 files at r3.
Review status: 6 of 8 files reviewed at latest revision, 9 unresolved discussions.


toxav/rtp.c, line 352 at r2 (raw file):

    }

    if (wkbl->work_buffer[0].frame_type == video_frame_type_KEYFRAME) {

chain with && instead of nested ifs


toxav/rtp.c, line 374 at r3 (raw file):

static struct RTPMessage *process_oldest_frame(Logger *log, struct RTPWorkBufferList *wkbl)
{
    if (wkbl->next_free_entry > 0) {

reverse condition and use early return


toxav/rtp.c, line 413 at r3 (raw file):

    if (slot == (wkbl->next_free_entry - 1)) {
        memset(&(wkbl->work_buffer[slot]), 0, sizeof(struct RTPWorkBuffer));
        wkbl->next_free_entry--;

this line is the the same in both branches, move outside conditional


toxav/toxav.c, line 885 at r3 (raw file):

                uint32_t frame_length_in_bytes = pkt->data.frame.sz;

                if (LOWER_31_BITS(frame_length_in_bytes) > 0x1FFFFFFF) {

invert condition so there isn't an empty if path


toxav/video.c, line 371 at r3 (raw file):

            (uint8_t)header_v3->pt == (rtp_TypeVideo % 128)) {
        LOGGER_DEBUG(vc->log, "rb_write msg->len=%d b0=%d b1=%d", (int)msg->len, (int)msg->data[0], (int)msg->data[1]);
        free(rb_write((RingBuffer *)vc->vbuf_raw, msg));

same line in both branches, move out of conditional


Comments from Reviewable

@iphydf iphydf force-pushed the video-fix branch 4 times, most recently from 82e068b to 81cd3ce Compare January 30, 2018 09:19
@iphydf iphydf force-pushed the video-fix branch 6 times, most recently from 25e0932 to 25766bb Compare February 6, 2018 14:36
@iphydf
Copy link
Member Author

iphydf commented Feb 6, 2018

Review status: 0 of 18 files reviewed at latest revision, 9 unresolved discussions.


toxav/rtp.c, line 153 at r1 (raw file):

Previously, iphydf wrote…

@zoff99 what should we call it?

Done.


toxav/rtp.c, line 443 at r1 (raw file):

Previously, iphydf wrote…

@zoff99 what should we do when this happens?

Done.


toxav/rtp.c, line 560 at r1 (raw file):

Previously, iphydf wrote…

That doesn't seem correct. But more interestingly, m_new should always be null here:

  • If m_new is null and slot is -2, it's free(null).
  • If m_new is not null, then regardless of session->mcb, it is set to null, so this is free(null).

It's always free(null). @zoff99 can you look into this?

Done.


toxav/rtp.c, line 621 at r1 (raw file):

Previously, iphydf wrote…

Same.

Done.


toxav/rtp.c, line 612 at r7 (raw file):

Previously, iphydf wrote…

It's either freed or passed to the callback mcb. The mcb callback becomes either a video or audio queue function, which either frees it or puts it into a jitter buffer (omg str8c, no unique_ptrs :().

Done.


toxav/rtp.c, line 630 at r7 (raw file):

Previously, iphydf wrote…

Done.

Done.


toxav/rtp.c, line 645 at r7 (raw file):

Previously, iphydf wrote…

Same. I've only looked for 3 minutes, but it seems ok.

Done.


Comments from Reviewable

@sudden6
Copy link

sudden6 commented Feb 6, 2018

:lgtm_strong:


Reviewed 1 of 13 files at r6, 5 of 19 files at r10, 4 of 14 files at r12, 8 of 8 files at r13.
Review status: all files reviewed at latest revision, 4 unresolved discussions.


Comments from Reviewable

@iphydf iphydf force-pushed the video-fix branch 2 times, most recently from 63f0e39 to 140a246 Compare February 8, 2018 17:26
@iphydf iphydf unassigned zugz Feb 8, 2018
@iphydf
Copy link
Member Author

iphydf commented Feb 8, 2018

@robinlinden, @endoffile78: this is ready for another round of review.

@jhert0
Copy link
Member

jhert0 commented Feb 11, 2018

:lgtm_strong:


Reviewed 8 of 8 files at r13.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.


Comments from Reviewable

@iphydf iphydf changed the title #2: Improve sending of large video frames in toxav. Improve sending of large video frames in toxav. Feb 11, 2018
@iphydf iphydf force-pushed the video-fix branch 3 times, most recently from db6de54 to 9fcd703 Compare February 11, 2018 21:50
@robinlinden
Copy link
Member

:lgtm_strong:


Reviewed 4 of 13 files at r6, 1 of 9 files at r9, 4 of 19 files at r10, 1 of 9 files at r11, 7 of 8 files at r13, 1 of 2 files at r15, 1 of 3 files at r16, 2 of 2 files at r17.
Review status: all files reviewed at latest revision, all discussions resolved.


toxav/rtp.c, line 49 at r17 (raw file):

{
    assert(allocate_len >= data_length);
    struct RTPMessage *msg = (struct RTPMessage *)calloc(sizeof(struct RTPMessage) + allocate_len, 1);

calloc generally takes num, size as arguments. This looks weird even if it does the same allocation.


toxav/rtp.c, line 300 at r17 (raw file):

    // frame data array.
    memcpy(
        (slot->buf->data + header->offset_full),

Unnecessary parentheses.


toxav/rtp.c, line 311 at r17 (raw file):

    slot->buf->header.received_length_full = slot->received_len;

    return (slot->received_len == header->data_length_full);

Unnecessary parentheses.


toxav/rtp.c, line 367 at r17 (raw file):

    LOGGER_DEBUG(log, "wkbl->next_free_entry:003=%d", session->work_buffer_list->next_free_entry);

    const bool is_multipart = (full_frame_length != incoming_data_length);

Unnecessary parentheses.


Comments from Reviewable

This change does not include the addition of VP9. We do that in a
separate pull request.

Changes:

* fix the video bug (video frames larger than 65KBytes) by sending full
  frame length in alternate header field
* improve video frame reconstruction logic with slots
* configure video encoder and decoder to be multihtreaded
* set error resilience flags on video codec
* change encoder and decoder softdeadline
@iphydf
Copy link
Member Author

iphydf commented Feb 11, 2018

Review status: all files reviewed at latest revision, 4 unresolved discussions, some commit checks pending.


toxav/rtp.c, line 49 at r17 (raw file):

Previously, robinlinden (Robin Lindén) wrote…

calloc generally takes num, size as arguments. This looks weird even if it does the same allocation.

Done.


toxav/rtp.c, line 300 at r17 (raw file):

Previously, robinlinden (Robin Lindén) wrote…

Unnecessary parentheses.

Done.


toxav/rtp.c, line 311 at r17 (raw file):

Previously, robinlinden (Robin Lindén) wrote…

Unnecessary parentheses.

Done.


toxav/rtp.c, line 367 at r17 (raw file):

Previously, robinlinden (Robin Lindén) wrote…

Unnecessary parentheses.

Done.


Comments from Reviewable

@zoff99
Copy link

zoff99 commented Feb 12, 2018

@robinlinden @iphydf
Unnecessary parentheses
i wished for those, for easier reading.

@iphydf iphydf merged commit 7213582 into TokTok:master Feb 12, 2018
@iphydf iphydf deleted the video-fix branch February 12, 2018 10:43
@iphydf iphydf modified the milestones: v0.2.x, v0.2.0 Feb 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants