general questions and thoughts (not in faq obv)

[idraper]

Hello, I am a dev who has some questions and general thoughts I'd like to talk about/ask here.

First things first - I disagree with everyone who says this is not possible. It's definitely possible from a technical standpoint and not a bad idea! However, that is not the same thing as saying this is feasible. There are many things that could get in the way of this getting accomplished from a real-world standpoint. The biggest central problem I see is a lack of experience regarding ML and treating it like a silver bullet. Having worked with ML quite a bit, I can tell you it can be very finicky; a real-world application like this requires more than just a general understanding of TensorFlow. Again, this is a technical problem that obviously can be solved, but this is more of a research problem, not a software engineering one.

People have brought up lots of valid concerns regarding things like cheat software being able "fake" being a human - but I disagree with this since that's exactly what ML would be good at. Detecting things not obvious to humans. However, it is not as simple as just giving it good data. You need people experienced with the models themselves, and very importantly, the math behind them. Everyone I've seen interested in helping doesn't have the necessary depth or experience for this kind of task. Basically, I think your main problem isn't a technical one like everyone is saying - it's a matter of finding the properly experienced data scientists and researchers willing to contribute (notice how I didn't say software developers :D). Sure, you can have people contribute things, but you need experienced people at the head to drive things. I have no idea who you guys are, but I don't think you have that kind of background (please correct me if I'm wrong). 2 months of research is fine for coming up with concepts, but nothing in terms of actually conducting research projects, which is really what this project is.

It worries me when I see statements like: "ML shouldn’t be underestimated" and "If you don't have any [experience], we still want you!" This is just not true - I think ML could be a good solution to this problem, but it is definitely not an easy problem. Second, you shouldn't just be accepting any and all developers. While a nice thought, it just isn't practical from an organizational standpoint. You certainly should be open to everyone, but it's not about just finding people to work on this - you need to find the right people. Otherwise, you will be overwhelmed with just managing people's PR's - a particular problem among junior devs and students (of who I see many...). Again, I think your central problem is that you have software engineers interested (which is part of the work) when your central issue is finding ML-experienced people. To be clear, I'm not against more inexperienced people helping, generally it's a win-win (experience for them and work done for you) - but I'm ONLY seeing these types of people (eg students), which is a problem.

TLDR; you need to focus on finding experienced ML researchers/engineers - not software engineers

---

Regarding the idea itself, I think it is a good one. However, I wanted to address some things regarding the longevity of the idea. Let's say it works fabulously, you are at a point where you can identify cheat software with great accuracy. Inevitably GAN's will be made for new cheating methods to avoid detection (eg an AI network designed to cheat while avoiding detection by being trained against yours). You will end up in a constant battle of catchup to defeat them. This is nothing new - Deepfake is experiencing this currently and it is a constant problem with cybersecurity. I bring this up to simply make the point that this doesn't magically fix the cheating problem. It simply will change the attack methods. To be clear - I still think this is a good solution because they will face the same challenges as you. However, it will inevitably come up if you are successful; it would be prudent to plan ahead for this.

---

Lastly, I just want to mention that this is a cool idea and has interesting problems. You need to have a better plan regarding setting up your project and organization. If you have a large number of people working on the project (as it looks like you might) this is critical to do BEFORE you start working on it.

I think this could be cool to work on - but I don't really have the time right now to be a central dev. Instead, I'd be happy to keep giving feedback and advice if you guys want. Hope this raised some thoughts on how you'll handle things moving forward.

[Brandons404]

I agree with most of this. Our biggest pain point and show stopper from the beginning has been trying to find someone with the background and skills required to lead development on something like this. And with a solid budget of $0.

I would like to reply to your point about longevity though, and the game of cat and mouse. Assuming it's built and works beautifully, I don't think the long term goal is to prevent any and all cheaters in the future. Sure it could catch a few, but tools for cheating that are built specifically to fly under detection of this tool will not be created until it exists. The whole idea of wanting to be able to scan video is that you can upload old videos. Theoretically any video between now and the beginning of time, to expose people who cheated previously. Some people might have the idea that this would be the end all solution but that isn't the point as far as I understand it.

[jaredb1011]

Good thoughts.

We actually went through very similar discussions to what you were mentioning, and we came to similar conclusions.
We've now stood up a dedicated prototype development team, and found some folks with relevant ML/AI experience to be a part of the team.

We'll be working on a prototype for the core functionality of the project, and once we have the core working, we'll open it back up more for others to help contribute for more general software development-type work.