[feature] FOOM detection.

[TristanTrim]

This seems like the sort of project that could accidentally produce a self-improving superhuman system. Does anyone on the project have an understanding of AI Alignment? Are there efforts to measure the potential for systems built with gorilla to FOOM?

[fire]

As non-staff, I am not sure how to measure something that doesn't exist.

If a self-improving superhuman system exists, we can use that to measure gorilla.

[TristanTrim]

@fire Thanks for engaging : )

As non-staff, I am not sure how to measure something that doesn't exist.

Are you claiming "potential for systems built with gorilla to FOOM" doesn't exist? Proof of that claim is exactly what I'm asking for.

If a self-improving superhuman system exists, we can use that to measure gorilla.

My understanding is, "we" can't use a self-improving superhuman for anything. It decides what to do based on whatever encoding of goals were in it when it went through self-improvement, and currently encoding goals is similarly not understood.

[fire]

Also what is FOOM?

[fire]

Also I am unsure of the timeline since OpenAI’s models are stronger and the promised work of apizoo integration isn’t done.

As far as I know GPT4 is not considered super human.

Also from a point of view of resources, we don’t have self fabricating ais yet that are also self improving.

[TristanTrim]

FOOM is an acronym: Fast Onset of Overwhelming Mastery. It refers to the hypothesized point when an AI system gets capable enough to modify its own software and improve itself, and in doing so improves it's ability to improve itself recursively.

GPT4 is in many ways superhuman. It's breadth of knowledge and speed at processing is vastly beyond any human. Only its depth and persistence of thought are lacking, and it may be possible to improve those with add-ons that don't require re-training the underlying model. So, it's my view that tools built with GPT4 are ambiguously FOOM capable and nobody has any real proof one way or the other rn.

The issue is that if an AI self improves, we don't know where it's capabilities will plateau. It may be far enough that it gains a decisive strategic advantage over all other planning agents (humanity) and then optimizes the universe for whatever (poorly) specified goal it was optimizing when it underwent self improvement.

Also, self fabrication is unnecessary for improving algorithms and utilization of existing hardware.

I understand that there is a common view that these things are far from human intelligence and far from dangerous, but they have been advancing so quickly in the last 10 years, and we really don't understand how they work, so I'd feel a lot more relaxed if I though people working on making them have agency and more capabilities were aware of the state of AI safety and AI alignment research.

Thanks again for your time reading my thoughts : )

[rajveer43]

can you eleborate why foom is needed

[TristanTrim]

Foom is something we want to avoid. If a system fooms, it may spread through cryptography and/or social exploits & cause unbounded harm by pursuing misaligned goals.

The fact that nobody knows what is required for a system to foom is a problem. Many people look at the state of AI today and think "this is obviously fine" while others think "we're already way past what is safe" and so we need to be getting people together on the same page about this and red teaming foom risk I think is a good step in that direction.