Video formats (on_server_handshake)

[aspann]

Hi,
I'm surprised why the payload.formats in "on_server_handshake()" has only SD and 720p profiles available, see:

DEBUG:xbox.nano.channel:VideoChannel server handshake payload: \s<XStructObj: None>(Container:
    protocol_version = 5
    width = 1280
    height = 720
    fps = 30
    reference_timestamp = 1539400773757
    formats = ListContainer:
        Container:
            fps = 30
            width = 1280
            height = 720
            codec = VideoCodec.H264
            rgb = None
        Container:
            fps = 30
            width = 960
            height = 540
            codec = VideoCodec.H264
            rgb = None
        Container:
            fps = 30
            width = 640
            height = 360
            codec = VideoCodec.H264
            rgb = None
        Container:
            fps = 30
            width = 320
            height = 180
            codec = VideoCodec.H264
            rgb = None)

I've also tried to rise the ClientVersion as well as correcting the resolutions in constants.py like:

class WindowsClientInfo(object):
    """
    Client Info for Windows device, used for LocalJoin messages
    """
    DeviceType = ClientType.WindowsStore
    NativeWidth = 1920
    NativeHeight = 1080
    DpiX = 96
    DpiY = 96
    DeviceCapabilities = DeviceCapabilities.All
    ClientVersion = 41
    OSMajor = 6
    OSMinor = 2
    DisplayName = "SmartGlass-PC"

That leads to the question, how the device determines the formats it's offering.
I'm feeling a bit lost, because I can't find anything corresponding in the docs (https://github.com/OpenXbox/smartglass-documentation/blob/master/docs/nano.md).

Needless to say, great work guys! 👍

[tuxuser]

Hey,

The resolutions / video formats the NANO protocol offers to the client are totally unrelated to the LocalJoin message from SmartGlass protocol!
The actual usage for LocalJoin by the console is -> sending content to render on the client - THEORETICALLY!
Only usage ever - Fallout 4 - but even there I am unsure if the companion app accounts for that info...

What you rather want to look into is this: https://github.com/OpenXbox/xbox-smartglass-nano-python/blob/master/xbox/nano/manager.py#L11

[aspann]

Wow, that was fast!

I've already changed that, but it seems not to have an impact on the offered resolutions.

...
    "videoFecType": "0",
    "videoFecLevel": "3",
    "videoMaximumWidth": "1920",
    "videoMaximumHeight": "1080",
    "videoMaximumFrameRate": "60",
    "videoPacketUtilization": "0",
    "videoPacketDefragTimeoutMs": "16",
    "sendKeyframesOverTCP": "false",
...

edit:
I've also changed the SDL resolutions.

[tuxuser]

It's likely a limitation in the protocol, I wasn't able to achieve higher resolutions either.
Keep in mind, encoding higher resolutions needs more processor power - and Xbox priority number 1 is speed! They likely don't want to risk any slowdowns ingame.

The way they achieve better quality (still on 720p tho) is increasing the bitrate - not the resolution.
In the current way of handling it, the nano client will take the highest bitrate automatically.

Reference: https://www.reddit.com/r/xboxone/comments/8ca0bl/xbox_app_increasing_streaming_quality_beyond_very/

[aspann]

Many thanks for those information.
I'll check and test it.

Another thing (resolution based) is, when activating the "VeryHigh" video quality in the channel.py. It seems to just display the topper-left part from the image. (like one fourth of the whole rendered image)

And I think that is the place where to rise the bitrate, right? (i.E. a new Enum like VideoQuality."Experimental")

[tuxuser]

Exactly!

About the cropped image, hmm, maybe the ffmpeg decoder needs more info to handle it sanely - PPS / SPS.

pyav (the ffmpeg python wrapper) is configurable towards that, but I was just using the basic wrappers - it surely is possible to optimize there.

[Schamper]

Or maybe the renderer is set to too low of a resolution https://github.com/OpenXbox/xbox-smartglass-nano-python/blob/master/xbox/nano/scripts/client.py#L53

[aspann]

@Schamper, good advise.

Sadly I already changed that to 1080p. I've also eddited: self._window, -1, (1920, 1080), (https://github.com/OpenXbox/xbox-smartglass-nano-python/blob/master/xbox/nano/render/video/sdl.py#L43)

still testing.

[aspann]

Tested it without really success..
I've rolled back everything to master, except the resolution settings in in client.py and manager.py.

My best try so far was:
Experimental = [70000000, 2, 60000, 1001, 30, 0]

As soon as I change the second value to 3 the cropping happens.
Is there a doc about this enum(or the VideoQuality itself), espacially about the 2nd - 5th value?

[tuxuser]

Official settings, provided by Microsoft's Xbox App config file (https://settings.data.microsoft.com/settings/v2.0/xbox/xboxapp):

GAME_STREAMING_VERY_HIGH_QUALITY_SETTINGS	"12000000,1080,60,59,0,10,40,170"
GAME_STREAMING_HIGH_QUALITY_SETTINGS	"8000000,720,60,59,0,10,40,170"
GAME_STREAMING_MEDIUM_QUALITY_SETTINGS	"6000002,720,60,3600,0,40,70,200"
GAME_STREAMING_LOW_QUALITY_SETTINGS	"3000001,480,30,3600,0,40,70,200"

Python lib (https://github.com/OpenXbox/xbox-smartglass-nano-python/blob/master/xbox/nano/enum.py#L99)

"""
6x uint32_t
"""
class VideoQuality(object):
    VeryHigh = [12000000, 3, 60000, 1001, 59, 0]
    High = [8000000, 2, 60000, 1001, 59, 0]
    Middle = [6000002, 2, 60000, 1001, 3600, 0]
    Low = [3000001, 1, 30000, 1001, 3600, 0]

Unsure how it correlates. Apparently there is no documentation as we have not figured out this piece...

[aspann]

Thanks for all the Information.

Currently I'm setting up a new VM to analyze the original protocol from the xbox-app (especially the codec-change). I'm closing this for now, maybe you can label this issue as "question".

(Maybe reopen this issue on new information)

cheers.

[tuxuser]

Ok, will do!
Reach out on Discord if you like

[tuxuser]

Source: https://github.com/MutoMagic/xstream/blob/master/xstream/Program.cs

// Modify standard config, if desired
/*
 * GAME_STREAMING_VERY_HIGH_QUALITY_SETTINGS: 12000000,1080,60,59,0,10,40,170
 * GAME_STREAMING_HIGH_QUALITY_SETTINGS: 8000000,720,60,59,0,10,40,170
 * GAME_STREAMING_MEDIUM_QUALITY_SETTINGS: 6000002,720,60,3600,0,40,70,200
 * GAME_STREAMING_LOW_QUALITY_SETTINGS: 3000001,480,30,3600,0,40,70,200
 * 
 * SETTINGS:
 * Unknown1,Unknown2,Unknown3,Unknown4,Unknown5,Unknown6,Unknown7,Unknown8
 * Unknown1 UrcpMaximumRate
 * Unknown2 VideoMaximumHeight
 * Unknown3 VideoMaximumFrameRate
 * Unknown4 ?
 * Unknown5 AudioBufferLengthHns if 0 use Unknown1
 * Unknown6 AudioSyncMinLatency
 * Unknown7 AudioSyncDesiredLatency
 * Unknown8 AudioSyncMaxLatency
 * 
 * refer to: https://github.com/OpenXbox/xbox-smartglass-nano-python/issues/7
 * standard: 10000000,720,60,?,0,10,40,170
 */
//config.UrcpMaximumRate = (int)(3.000001 * 1000000);// 1后面6个0
//config.VideoMaximumHeight = 480;
//config.VideoMaximumFrameRate = 30;
//config.Unknown4 = 59;
//config.AudioBufferLengthHns = 0;
//config.AudioSyncMinLatency = 40;
//config.AudioSyncDesiredLatency = 70;
//config.AudioSyncMaxLatency = 200;
string quality = GetSettingString("GAME_STREAMING_AVAILABLE_QUALITY_SETTINGS");
if (mapping.Length == 2 && mapping[1].Length != 0)
    quality = mapping[1];

config.UrcpMaximumRate = GetConfigurationInt(quality, "UrcpMaximumRate");
config.VideoMaximumHeight = GetConfigurationInt(quality, "VideoMaximumHeight");
config.VideoMaximumFrameRate = GetConfigurationInt(quality, "VideoMaximumFrameRate");
//config.Unknown4 = GetConfigurationInt(quality, "Unknown4");
config.AudioBufferLengthHns = GetConfigurationInt(quality, "AudioBufferLengthHns");
config.AudioSyncMinLatency = GetConfigurationInt(quality, "AudioSyncMinLatency");
config.AudioSyncDesiredLatency = GetConfigurationInt(quality, "AudioSyncDesiredLatency");
config.AudioSyncMaxLatency = GetConfigurationInt(quality, "AudioSyncMaxLatency");

if (config.AudioBufferLengthHns == 0)
    config.AudioBufferLengthHns = config.UrcpMaximumRate;

// 由于小数点向上进位，因此误差 +-1 的情况下，永远满足最小分辨率16:9
config.VideoMaximumWidth = (int)Math.Ceiling(config.VideoMaximumHeight / 9.0 * 16);