Video Analysis Ideas #13
Replies: 9 comments 11 replies
-
|
I've seen this sort of camera tracking in blender from this video of realistic minecraft: https://www.youtube.com/watch?v=Z6PglunJFFg. This may help. |
Beta Was this translation helpful? Give feedback.
-
|
Also, if we are ultimately going to extrapolate mouse position data, then won't specific mouse dpi come into play? |
Beta Was this translation helpful? Give feedback.
-
|
Definitely. Factors like FOV, dpi, sensitivity, mouse accel, etc all come into play. That's why we need a lot of varied data, and the ability to pull out different variables/factors from the video |
Beta Was this translation helpful? Give feedback.
-
|
yes, but is there really a way to know dpi or specific mouse acceleration settings? Maybe if we observe the mouse by itself on the screen we may be able to tell... |
Beta Was this translation helpful? Give feedback.
-
|
As for a hard DPI number, no. Something more accomplishable would be running something like a Simultaneous Localization and Measurement algorithm on this input data. From there we can extract the camera motion from one frame to the next. The next step would be to look for the levels of precision with regard to camera motion. There may be some stats we can run to get an estimate of how sensitive their input setup actually is based purely upon user input. Assuming that camera parameters don't change from player to player, that data may be usable. |
Beta Was this translation helpful? Give feedback.
-
|
Hey. Im a Photo/Videographer with a background in Programming and i can tell you right now that general video analysis will run into bitrate and artifact issues, especially on Stream footage which is always relatively low bitrate. Beyond that, this video analysis will have to take into account and deal with any non-game elements like popups, webcams, various overlays and even Emoji spam. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah those are all good points. I think to start out we're going to use really clean footage with no overlays or anything weird, and we'll have to gradually introduce more clutter factors and work solutions |
Beta Was this translation helpful? Give feedback.
-
|
Artifacting is my biggest concern. In any piece of gameplay, the center area of the screen near the reticle is generally free of obstructions. We'll need to look at the minute movements of the mouse once a target is already in the crosshairs, not large panning movements. |
Beta Was this translation helpful? Give feedback.
-
|
For guidance we could look into what CCTV Face-recognition software does, as many of the same issues are dramatically worse there (camera quality mostly), and in terms of AI, recognizing a face is dramatically more difficult. |
Beta Was this translation helpful? Give feedback.
-
|
I posted this in another reply but what if we were to allow the user to highlight the target that would outline the object similar to creating a mask for an object and then a second one that would highlight the gun’s reticle. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding the ML part We need to define what are the easiest cases in which we can detect cheating and hard cases, this will give us some idea of the layers of execution of the ML system This means some models get executed first to detect easy cheating cases (ex. extra information displayed in the screen tags/position/boxes/wall hacks) We also need to identify how to preprocess the data to train, the first thing is to look at several videos and identity possible windows (clips) in which there is a clear evidence of cheating then we can proceed to check how can we create pipelines for labeling and data storage |
Beta Was this translation helpful? Give feedback.
-
|
What if we allow training to be done on the flash so for example using a selection tool to identify the target and then a selection tool to identify a gun’s reticle that would allow us to track cursor movement on a target. |
Beta Was this translation helpful? Give feedback.
-
|
Seeing how this step processes video footage for the AI classification part. It's really important that the processed data is precise and accurate. Obviously the plan is that a user will just submit footage for analysis. But it might be ideal to start with footage that had the mouse movement data recorded by the user. That way, we have example mouse data to test against, and proper data to train the AI with. The problems with this are:
But as a first step, it's worth at least considering. This can be done either by just using another program to record mouse data, but refresh rate differences and syncing up the times comes in. However, you can also try parsing replay data from games that support replays. Once again, as a starting point it may help. Simplify the first build and all. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with this, especially during earlier parts of development, a lot of focus should be put on ensuring the "Hacking" and "Legit" videos are what they say they are, as development will be tuning parameters to ensure the results match. A bad actor can really mess up development before it really gets its legs. |
Beta Was this translation helpful? Give feedback.
-
|
The only thing with this is what if somebody’s actually more accurate than your initial test subject they could potentially be flagged as cheating even though they’re actually very good at the game. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if this is helpful at all, but I had an idea on one way to help tell if someone is hacking. From what I can tell most of the software that is out there right now focuses in on a specific identifiable point on the character that you are attacking, lets say the middle of the chest, any human might be able to hit the chest of someone but a computer would be able to hit the same spot of the chest every time. So if you basically map the model with coordinates then see how close the pattern of shots is to the first shot at that target, and if there are multiple kills, you can check against the other models the player shoots at as well. |
Beta Was this translation helpful? Give feedback.
-
|
just had a quick thought. games with replay systems will help a huge amount in this way. take for example CSGO or R6s. the ability to take as many replays from as many people that play the game and be able to comb through every persons POV is an incredibly useful one, and just figured i would point this out as it could be a good start as to where we could get mass quantity of hacker/assisted vs human gameplay footage |
Beta Was this translation helpful? Give feedback.
-
|
I find a different way called image stabilization. One could use that to reverse the calculations on how much stabilization is being used. Here’s a link with some code. |
Beta Was this translation helpful? Give feedback.
-
|
My thought was this is if we can use an image stabilization and track how much is actually being changed then we could use those variables to determine which one is hacking which one is not. For example if we see only a three pixel difference between up and down and left and right versus five and six then we know that the person doing one and two is actually getting computer-assisted. |
Beta Was this translation helpful? Give feedback.
-
|
Is there any way to get the info on how much a player's mouse actually physically moved during a game? Probably not, right? But yeah if you could, you could figure out the exact amount/direction of on-screen movement that X amount of physical mouse movement should cause for a legit player. Then if there is some sort of aim assist / aim lock that's helping to pull the aim in some direction, it wouldn't fit with the normal numbers, and would prove there's something else contributing. Ah I wish COD would somehow make that data available, like at least for 24 hours after the game ends or something |
Beta Was this translation helpful? Give feedback.
-
Current Status
So far, all I've worked on is using a technique called Optical Flow in the OpenCV libraries to calculate the motion of objects and pixels across a screen in gameplay footage. (what was in the basicallyhomeless vid). That's the code that's currently on the github.
The idea is that there may be a way to use that motion data and figure out how the player camera (aim) is changing throughout a video.
What we need
Research and test development of how to use optical flow to get some sort of aim vector/data.
Research on how 3D tracking works in video editing software. Are there existing methods for figuring out how a camera is moving through a scene by analyzing the video?
Please post any ideas or thoughts in this discussion!
Beta Was this translation helpful? Give feedback.
All reactions