Running this project daily doesn't make sense if GPT-4 is not being constantly updated

[simonw]

This project seems to be based around the idea that the GPT-4 model will get constant updates.

GPT-4 was trained over several months against a fixed pool of training data with a September 2021 cut-off date. It's not being re-trained on a regular basis.

GPT-4 output is also random, so any differences in unicorn pictures are far more likely to result from that randomness than changes to the underlying model.

Something that might make more sense: run this project once a month, and have it generate 16 unicorn images. You can then compare that 4x4 grid over time.

Bonus: you could use the API to render a 4x4 grid using GPT3 and GPT3.5 first.

You could even watch the https://api.openai.com/v1/models API and render unicorns for every model there - then keep an eye out for new models listed there.

Here's the current output of that endpoint: https://gist.github.com/simonw/07148ee63e127959507c231401b144b0

[leikareipa]

Is enough about the GPT-4 system architecture known that we can be sure about the frequency of updates to whatever comes out? If it's already known then no point, but otherwise can't hurt to get some baseline.

I think it could also be an experiment run in private, so there's less of a chance of pollution.

[simonvizzini]

Why though? I guess nobody outside of OpenAI knows how often the GPT-4 model is updated. What if the model is updated multiple times during one month? Then the gradual improvements won't be really visible by this test. I think daily updates make sense if the frequency of updates is not known to get a better picture of the improvements.

I agree though that it would be better to generate more than 1 unicorn image, random noise might skew the result a lot otherwise.

[adamkdean]

I've updated the accompanying blog post to clarify that the daily variance is to be expected but we'll also be able to see, over time, whether this changes with model updates too. Currently, we're seeing a lot of randomness, but with updates, we may start to see a general consensus of what a unicorn looks like across various samples.

I could generate more than one per day, but I expect we'll end up with the same amount of variance just in greater quantity. With this being a long term project, I think the sample size will grow large enough in time.

I guess nobody outside of OpenAI knows how often the GPT-4 model is updated. What if the model is updated multiple times during one month? Then the gradual improvements won't be really visible by this test. I think daily updates make sense if the frequency of updates is not known to get a better picture of the improvements.

Exactly. It's expected that updates will become more iterative, and this project will track those changes (within the gpt-4 model family) automatically. The specific model used is recorded with each output, so we can see changes.

I appreciate the input @simonw, this isn't the perfect science experiment, but I'm expecting it'll be a lot more exciting to review & discuss a year from now.

[adamkdean]

Closing this issue