HellaSwag scores using the perplexity tool

[klosax]

The HellaSwag test

The test is done by using examples of sentences that makes sense and others which do not make sense. The sentences are presented to the model in groups (tasks) of four with the same start but different endings. One of the endings in each group is labeled as the one that makes most sense. The probability of the model to predict the different endings are computed and if the labeled ending got the highest probability it it considered a correct prediction. The resulting accuracy score is the percent of groups with correct predictions.

More info can be found in the paper https://arxiv.org/abs/1905.07830

In order to compute the HellaSwag scores you need to download the datafile here: klosax/hellaswag_text_data.

Simple usage that runs the test on 400 random tasks in the datafile:
./perplexity --hellaswag -f hellaswag_val_full.txt -m modelfile.gguf

For a more accurate test you can specify the number of tasks to use in the computation:
./perplexity --hellaswag --hellaswag-tasks N -f hellaswag_val_full.txt -m modelfile.gguf

The table below is showing the score (0-shot acc_norm) for various models using 400 randomized HellaSwag tasks. The numbers in the "Leaderboard" column are taken from HuggingFace Open LLM Leaderboard. Note that the Leaderboard scores are about 1.5 higher because they are 10-shot. The "Params" column is the total number of elements in the model file and "Train tokens" are the total number of tokens the model was trained on.

Model	F16	Q8_0	Q5_1	Q4_0	Q2_K	HF Leaderboard	Params 10^9	Train tokens 10^9
LLaMA-7b	75.25	75.25	75.75	73.75	69.25	77.80	6.74	1000
LLaMA-13b						80.90	13.02	1000
LLaMA-33b						84.70	32.53	1400
LLaMA-65b						86.10	65.29	1400
LLaMA2-7b	77.50	77.50	77.00	77.25	72.50	78.60	6.74	2000
LLaMA2-13b	78.50					82.10	13.02	2000
LLaMA2-70b						87.30	68.98	2000
OpenLLaMA-3b					n/a	62.60	3.43	1000
OpenLLaMA-7b	70.75	70.75	70.50	69.25	65.50	67.80	6.74	1000
OpenLLaMA-13b					n/a	71.10	13.02	1000
OpenLLaMAv2-3b	69.75	69.75	68.25	68.00		71.60	3.43	1000
OpenLLaMAv2-7b	75.00	75.25	75.00	74.00	71.75	72.20	6.74	1000
Falcon-7b	76.75	76.75	76.25	74.50	n/a	78.10	7.22	1500
Falcon-40b					79.25	85.30	41.84	1000
MPT-7b						77.57	6.65	1000
MPT-7b-8k	75.25						6.65	1500
MPT-30b						82.40	29.96	1000
GPT-J-6b	64.50					67.60	6.05	402
Pythia-70m-dd	26.25	27.25	23.50	21.00		27.20	0.07	300
Pythia-160m-dd	30.75	31.00	28.50	28.00		31.40	0.16	300
Pythia-410m-dd	40.25	40.50	40.00	38.00		41.30	0.41	300
Pythia-1b-dd	49.00	49.25	49.50	48.25		49.70	1.01	300
Pythia-1.4b-dd	54.25	53.50	54.50	53.25		55.00	1.41	300
Pythia-2.8b-dd	58.75					60.70	2.78	300
Pyhtia-6.9b-dd						67.00	6.86	300
Pythia-12b-dd	68.25					70.30	11.85	300
GPT2	28.50	29.75	26.75	31.00		31.60	0.16	?
GPT2-Medium	39.25	40.75	37.75	37.25		40.20	0.41	?
GPT2-Large	43.00	43.25	43.00	44.25		45.60	0.84	?
GPT2-XL	53.75	53.75	53.00	52.50		51.40	1.64	?
Random accuracy						25.00
Human accuracy						> 95.00

The second part of the table contains models not yet supported in llama.cpp, but support may be added in the future.

The HellaSwag scores are correlated to the number of model parameters:

The 400 task 0-shot HellaSwag scores are highly correlated to the OpenLLM Leaderboard 10-shot HellaSwag scores:

(This post will be updated)

[gjmulder]

A wider view of progress over time using the HF leaderboard data. I curled the lastModified date from HF for each model on the HF leaderboard and plotted the Average score per model size. You can see the original FB base Llama model scores from late Feb as the earliest dots on the far left.

Conclusion: Continuous and continuing progress upwards with the 7B models improving twice as fast as 65B, likely due to compute availability.

[Green-Sky]

did you combine llama v1 7B with v2 7B ?

[klosax]

If you take the log(n_params) and compare it to the hellaswag scores, you can get a picture of how much models of different sizes can actually learn common-sense reasoning. Calculating the linear regression at different points in time, we could possibly use the function constants to plot a general chart of how the hellaswag capacity of LLMs changes over time.

[gjmulder]

did you combine llama v1 7B with v2 7B ?

Yes.

The plot was intended to provide a six month view of the consistent "small-model" incremental LLM progress as a function of model size. Small, specialist models can now outperform generalist models for specific use cases, e.g. coding, math, creative writing, etc. These models can be iterated and improved on in weeks, and deployed easily using inference frameworks like llama.cpp.

Compare this to Bard, GPT, etc. where progress has tended to be discontinuous, i.e. prior to this year, every year or so, a large tech company has moved SOTA forward with typically some innovation and a lot of scaling - at the cost of 10's of millions of dollars of compute.

The recently leaked GPT-4 architecture and compute costs indicate that scaling is probably not the way forward. GPT-4 is a somewhat larger GPT-3, and then scaled horizontally by using a Mixture of Experts (MoE), while still costing significantly more in compute to train and run. If MoE is the "way forward" (unlikely), we can conceivably run a MoE of specialist smaller models.

[klosax]

The scores of the LLaMA models on the Open LLM Leadboard have been recalculated (several times) due to tokenizer problems. EleutherAI/lm-evaluation-harness#531 (comment)

Problems with the tokenizer may be one of the reasons why the scores in our HellaSwag test is lower than the Leaderboard scores. The other reason is that we are currently doing 0-shot tests and not 10-shot. There is some discussion about this here #2389.

[gjmulder]

The scores of the LLaMA models on the Open LLM Leadboard have been recalculated (several times) due to tokenizer problems. EleutherAI/lm-evaluation-harness#531 (comment)

Yes, I seem to remember this was a significant problem with the OpenLLM family of base models. Of course the HL benchmark inconsistency has educated people on how important the tokenizer is in terms of model output quality.

I'm plotting the HL average across all the EleutherAI benchmarks and hopefully by looking at 400 or so models a lot of the noise from problematic tokenization and other issues is averaged out.

What will be interesting is comparing scores in another 6 months time and seeing if the trends for each model size continue improving.

[ianscrivener]

Great work @klosax , @gjmulder and team 🏆

[ikawrakow]

The HellSwag calculation is broken on current master: I get a score of 71.00 after 400 tasks for LLaMA-v2-7B (see attached output). The change from the score of 77.5 we find in the table above occurred after PR 6381d4e.

I have checked that the result is also different for LLaMA-v1-7B (68.50 vs 75.25 after 400 tasks).

broken.txt

[klosax]

Is the perplexity on the model you used different?

[klosax]

All my results in the table above was done using cublas without offloading before the custom cuda mat mul was made default in PR #2683. Test disabling the mmq by using the -nommq parameter.

[ikawrakow]

@klosax When someone says that commit X breaks something, what is meant by that is that in commit X-1 the thing that got broken still works. This is exactly how I mean it. In commit dadbed9, which is the one before 6381d4e, I get the same scores as posted in the table above. In commit 6381d4e, HellaSwag scores are significantly lower as described above. 6381d4e is a massive change that brings in the GGUF format. Yourself, @ggerganov, @goerch , @slaren, @KerfuffleV2, etc., participated. Unless you think that I'm confused enough to run dadbed9 with -nommq and 6381d4e with -mmq, the change cannot possibly depend on -mmq vs -nommq. But even if you thought that I was sufficiently confused to do that, if you paid attention to the log I provided with my comment, you would have seen that I'm running with the fp16 model, where -mmq/-nommq is completely irrelevant. Now, I can go through the +10090/-3025 lines changed in 6381d4e myself to try to sort out what could have broken the HellsSwag scores, but typically the expectation is that when something gets broken by a specific commit, the author(s) of that commit do that instead.

[KerfuffleV2]

but typically the expectation is that when something gets broken by a specific commit, the author(s) of that commit do that instead.

The GGUF commit seems like a bit of a special case since it involved a lot of changes from different authors and also administrative decisions like how tokenizing is performed, vocabulary stored, etc. I think it would be really hard to solve a regression in this specific case by going through the changes and trying to read them.

Also, can't blame me in this case. :) I'm an "author" of the patch technically but all I did was contribute a stand-alone script for converting GGML files to GGUF and I doubt you're testing with a model converted using that.

Something related to the tokenizing/vocab changes seems like it's probably most likely to be causing this. If I was debugging it, I'd probably try to run hellaswag on a specific set of test cases and dump exactly the tokens are getting sent to the model to prompt it. It should be possible to confirm or rule out tokenizing stuff that way (or other similar issues like BOS getting included or not).

[klosax]

Sorry, I must be the one who is confused by the commit numbers and the problems with mmq in Falcon. Anyway,. the GGUF commit was massive and one thing that comes to my mind that could be related to the current HellaSwag scores is the tokenizer fixes from PR #2549.

[ikawrakow]

Let me try again.
Commit dadbed9: HellaSwag works
Commit 6381d4e: HellaSwag broken, results 4-5 percentage points lower
Current master: HellaSwag broken, results 4-5 percentage point lower, same as 6381d4e.
Logical conclusion: unrelated to any tokenizer changes that have happened since or before 6381d4e, breakage is caused by whatever was changed in 6381d4e.

[KerfuffleV2]

Logical conclusion: unrelated to any tokenizer changes that have happened since or before 6381d4e, breakage is caused by whatever was changed in 6381d4e.

Yes, and there were major tokenizer changes in 6381d4e and the way vocabulary gets stored/processed for the model files also changed a lot. That's what I was talking about.

If you can compare the token ids that get sent to the model before and after the GGUF patch you can either confirm or rule out those changes as causing this effect.

[klosax]

Yes, in commit 6381d4e (PR #2398 aka ggufbranch) the PR #2549 and #2668 was added among others, which brought tokenizer fixes.

[ikawrakow]

Yes, and there were major tokenizer changes in 6381d4e and the way vocabulary gets stored/processed for the model files also changed a lot. That's what I was talking about.

Excellent! This gives me the opportunity to rant against the policy of sqaush-merging massive changes to the code base, with 6381d4e being a great example why that is not good. Do I want to have the 200+ commits that lead from dadbed9 to 6381d4e in the history? Definitely not. Would it have been better to restructure the 200+ commit in #2398 into X logically related commits instead of squash-merging? I think yes. Then it would have been easy to find the culprit via git-bisect. But the way it is, I continue to encourage the authors of #2398 and @ggerganov to find the problem.

[KerfuffleV2]

It's still possible to see the individual unsquashed commits that made up the pull: https://github.com/ggerganov/llama.cpp/pull/2398/commits

Would it have been better to restructure the 200+ commit in #2398 into X logically related commits instead of squash-merging? I think yes.

That's certainly true, but it's way easier said than done. There are a large number of commits that are interdependent, and even depend on specific versions of other commits. Maybe it would be possible to go through everything and bundle sets of changes into logic chunks but it would be both difficult and time consuming.

No one volunteered to do that, and let's face it most people really don't find that kind tedious administrative task very fun. As open source contributes, we mostly contribute the stuff that we enjoy working on or that scratches an itch, etc. Just coordinating the changes in the GGUF pull was a pretty big task (and the transition was surprisingly smooth). Should we say Mr. GG should have just buckled down and put in the work to try to organize the pull better once it reached a mergeable state? Or if not him, who should have been assigned to do it?

edit: Just want to add, I don't mean this as a rebuke or criticism. I'm also not saying it's unreasonable to be frustrated in this situation, it's definitely not ideal. My point is only that there isn't really a simple/easy solution to the problem.

[ikawrakow]

So, the HellaSwag score change after #2398 is due to this line:

llama.cpp/llama.cpp

Line 3030 in bae5c5f

std::string result = "\xe2\x96\x81";

If I comment it out, I recover pre-GGUF scores. If I replace it with

    std::string result;
    if (!text.empty() && text.front() != ' ') result = "\xe2\x96\x81";

I get a score very close to pre-GGUF but not quite the same (see #2805)

[ikawrakow]

Here are some HellaSwag 10-shot scores for LLaMA-v2-7B. The test cases along with the 10-shot context added that I have used are attached (see at the end of the post). I tried a few variations and this version gave the highest score for the fp16 model: 78.164 +/- 0.586. This is short of the 78.6 posted on the HF leader board for LLaMA-v2-7B, but much better than the 75.80 +/- 0.43 one gets with the 0-shot evaluation one gets with hellaswag_val_full.txt from https://github.com/klosax/hellaswag_text_data after running all 10042 tasks in the dataset.

The dataset consists of 5001 tasks. These are the tasks from activities that have training data in the HellaSwag training dataset, and the context added before each task was taken from there. I verified that for 0-shot evaluation with this 5001-task subset I get the exact same 0-shot HellaSwag score of 75.8 as with the full 10042-task validation dataset.

The following table shows the difference in HellaSwag score for different quantization types. If you want to have the actual HellaSwag score for a given quantization, just use 78.164 + Difference to fp16. E.g., for Q2_K, we have HellaSwag score = 78.164 - 2.060 - 76.104.

Quantization	Difference to fp16
Q6_K	+0.060 +/- 0.111
Q5_K_M	-0.280 +/- 0.144
Q5_K_S	-0.460 +/- 0.148
Q4_K_M	-0.460 +/- 0.175
Q4_K_S	-0.780 +/- 0.190
Q3_K_M	-1.420 +/- 0.255
Q2_K	-2.060 +/- 0.312
Q5_0	-0.200 +/- 0.162
Q5_1	-0.220 +/- 0.148
Q4_0	-0.960 +/- 0.209
Q4_1	-1.000 +/- 0.224

Here is a plot of the data in the above table. The x-axis is 100 * (perplexity(quantization) - perplexity(fp16))/perplexity(fp16). We see that there is a very strong correlation between HellSwag score and perplexity.

hellaswag_10shot.txt.gz

[klosax]

I tried a few variations and this version gave the highest score for the fp16 model:

The dataset consists of 5001 tasks. These are the tasks from activities that have training data in the HellaSwag training dataset, and the context added before each task was taken from there.

How did you choose the tasks? The tasks should be fully randomized and not hand picked for a specific model. Does this 10-shot test work equally well on other models like Falcon?

IMO the training data should not be used at all when testing. To measure how good the model generalizes outside of training, only the validation dataset should be used as a test target.

The OpenLLM 10-shot are highly linearly correlated to the 400 task 0-shot. The 0-shot being about 1.5 lower then the 10-shot.

We see that there is a very strong correlation between HellSwag score and perplexity.

The strong correlation is only valid for models trained on the exact same dataset. LLaMA-7b perplexity is much lower than Falcon-7b on the wiki.test.rawdataset, but the models generalizes about the same on other tests like HellaSwag.

[ikawrakow]

How did you choose the tasks? The tasks should be fully randomized and not hand picked for a specific model. Does this 10-shot test work equally well on other models like Falcon?

As explained, those 5001 tasks out of the 10042 tasks in the HellaSwag validation dataset were taken where training data of the given activity were available. The following activities did not have training examples:

Clean and jerk
High jump
Playing harmonica
Sharpening knives
Having an ice cream
Cutting the grass
Washing face
Gargling mouthwash
Running a marathon
Ice fishing
Playing violin
Layup drill in basketball
Personal Care and Style
Family Life

Because I expected that someone may object to selecting these 5001 tasks, I specifically pointed out that the 0-shot HellaSwag score for these 5001 tasks is exactly the same as the 0-shot HellaSwag score of the excluded 5041 tasks. I.e., I did not add any kind of bias into the evaluation by picking these 5001 tasks.

Does this 10-shot test work equally well on other models like Falcon?

Have not tried. The calculations with the full data set take a very long time. Evaluations with just 400 tasks are basically useless as statistical uncertainty is in the range of +/-2 (just look at the 77.5 score posted for LLaMA-v2-7B in the above table and compare that to the final score of 75.8 after 10042 0-shot tasks, where the statistical uncertainty still is +/- 0.4). Perhaps you could try to run Falcon-7b with the full evaluation dataset (0-shot plus the 10-shot version I provided)?

IMO the training data should not be used at all when testing. To measure how good the model generalizes outside of training, only the validation dataset should be used as a test target.

And how do people do 10-shot HellaSwag in your opinion? The 10 examples added to the context along with the actual task must come from somewhere. In general, as long as the data being used for prediction on a given validation/test dataset does not contain traces of that validation/test dataset, one can do whatever one likes with the training data. One can wipe the floor with it, cook coffee with it, paint the walls with it, you name it. And, among all the things one can do with the training data, one can add it as a context when predicting the continuation of a sentence in the HellaSwag test. Saying that one cannot use the training data when running inference on a dataset that did not participate in training is equivalent to saying that we can not use inference to test a model because, you know, the training data is encoded and present in some form in the model weights. If still in doubt, I can give you an example from $work that is completely unrelated to LLMs. The task is to generate a CT image from a given MR image of a patient. One of the algorithms we use is based on deformable image registration. Basically, you have a training dataset consisting of registered MR-CT image pairs of a set of patients. To generate the CT image of a new patient, one picks the K most similar patients from the training data, runs deformable image registration between the selected K training MR images against the new MR image, and uses the resulting deformation fields to warp the corresponding training CT images onto the MR image of the new patient. So that, unlike LLMs, where training data is not present directly but is encoded in some way in the model weights, here we use the training data directly to predict a CT image of an yet unseen patient. To test how well this algorithm works, we separate a given set of patients into a validation dataset (so they are not used as training data), and then evaluate the accuracy of the generated CT images by comparing with the known CT images of these validation patients. So, then, based on your comment, we shouldn't be doing that? And, the professors at the university and text books that thought us that doing this is OK were all confused?

The OpenLLM 10-shot are highly linearly correlated to the 400 task 0-shot. The 0-shot being about 1.5 lower then the 10-shot.

Yes, I saw the plot, and yes, I see there is correlation. But I also see in the very same plot that two models with the same 0-shot-400-tasks score can differ by 5+ points from their respective OpenLLM 10-shot score (and from the linear fit of the data). 5 HellaSwag points error is just too big to disambiguate between modern models, so not really useful for anything but distinguishing a modern model from GPT-2.

The strong correlation is only valid for models trained on the exact same dataset. LLaMA-7b perplexity is much lower than Falcon-7b on the wiki.test.rawdataset, but the models generalizes about the same on other tests like HellaSwag.

Sure. But in the table above I see columns for fp16 and various quantization types for the same model. Which implies that at some level this table attempts to use HellaSwag scores as a measure for quantization quality of a given model. The graph above tells us that, other than having spent 20X longer to evaluate, we learn nothing new about quantization quality from HellaSwag that we didn't already know from the 20X shorter perplexity calculation.

[klosax]

And how do people do 10-shot HellaSwag in your opinion? The 10 examples added to the context along with the actual task must come from somewhere.

I do not know how it is supposed to work. That is why I settled on doing 0-shot only using the validation dataset. Maybe you are right that > 0 shot is supposed to be using training data to make the model better understand the activity/context before predicting the endings.

Yes, I saw the plot, and yes, I see there is correlation. But I also see in the very same plot that two models with the same 0-shot-400-tasks score can differ by 5+ points from their respective OpenLLM 10-shot score.

Perhaps the problem here is also the statistical uncertainty when evaluated using only 400 tasks.

The only variable that correlates well with HellaSwag over different models and sizes seems to be the (log of) the number of model parameters.

[klosax]

The datafile for 10-shot contain <s> and </s> which I guess is supposed to be resolved to BOS and EOS tokens (using LLaMA models). But it does not seem to work:

main: prompt: '<s>Hello</s>
     1 -> ''
   529 -> ' <'
 29879 -> 's'
 29958 -> '>'
 10994 -> 'Hello'
   829 -> '</'
 29879 -> 's'
 29958 -> '>'

I guess tokenizers should not resolve the special tokens by default, but I think we should have that option to get things like this to work properly.

[qnixsynapse]

Tbh, I don't understand this test.
For LLaMA2-chat-13b-q4_k_m.gguf quant:
Some tasks got full 100 while some drop to 50. It got stable on 77-78 which is nearly same as its fp16 counterpart.

The test ran for 400 tasks.

Edit: It looks something like this:

[ikawrakow]

In each task you have a sentence plus the beginning of the next sentence. There are 4 possible continuations given, one of the 4 is the expected continuation. The algorithm evaluates the probability of the 4 continuations and picks the one with the highest probability. If the selected continuation is the expected one, +1 is added to the score. After evaluating N tasks and observing n correct continuations, the HellaSwag score is 100 * n/N. When you run the perplexity tool with --hellaswag it will output the current score after each task. For small numbers of tasks the printed score is subject to large statistical fluctuations. As you increase the number of tasks the score stabilizes. The statistical uncertainty after N tasks is 100 sqrt(n/N * (1 - n/N)/(N-1)). So, for n/N ~ 0.75, you have an uncertainty of +/- 4 after 100 tasks. It does not make sense to look at the score for anything less than 100 tasks.

[qnixsynapse]

I see. Thank you.

[ikawrakow]

@klosax

Does this 10-shot test work equally well on other models like Falcon?

I ran Falcon-7b now. I get 10-shot HellaSwag = 77.904 after 5001 tasks, which is almost as good as the score of 78.13 posted on the OpenLLM leader board. So, I guess, we could say that it works for Falcon as well as, if not better than, LLaMA models.

Interestingly enough, in 0-shot evaluation Falcon-7b beats LLaMA-v2-7b: 76.97 vs 75.8 after 10042 tasks.

I used a different 10-shot data file, which gets 10-shot HellaSwag = 78.24 for LLaMA-v2-7B (so very slightly better than the 78.16 I had before for LLaMA-2). I'm attaching it below. What is the difference between the two versions? In the original version that I used in my post above, I used the 10 longest examples from the training data that belong to the given activity. In this version, the training examples are selected randomly using a probability proportional to the example length squared.

hellaswag_10shot_v2.txt.gz

[klosax]

The <s> should be tokenized to token id 1, but it does not, so we need to search the input for those and segment the tokenization and insert the ids.

[KerfuffleV2]

But token id 1 in Falcon models is something different:

    {
      "id": 1,
      "content": ">>ABSTRACT<<",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false,
      "special": true
    },

Token id 11 is both BOS and EOS:

    {
      "id": 11,
      "content": "<|endoftext|>",
      "single_word": false,
      "lstrip": false,
      "rstrip": false,
      "normalized": false,
      "special": true
    }

Pasted from https://huggingface.co/tiiuae/falcon-7b tokenizer.json

[klosax]

Those should be saved by the conversion scripts to the gguf model file and be accessible in C (llama.cpp) by using the variables vocab.special_bos_id and vocab.special_eos_id. The ids are the same as those displayed when the model is loaded.

Falcon does not seem to have a differentiation of BOS and EOS like LLaMA models does.

[KerfuffleV2]

If you convert using #2842 does it help? Assuming you're using a venv you'll need to pip install --upgrade ./gguf-py after switching to that fork.

[klosax]

No the problem is in the llama.cpp tokenizer code. It will not tokenize the special tokens string values to the special token ids and I think it should not normally do that since <s> could be a reference to something else like html codes. What is needed is a option to the tokenizer in llama.cpp to tokenize these for uses like the we are doing here.

[klosax]

Input: <s>First sentence</s><s>Second sentence</s>
We would have to do the tokenization like this:

add vocab.special_bos_id
add result of tokenize "First sentence"
add vocab.special_eos_id
add vocab.special_bos_id
add result of tokenize "Second sentence"
add vocab.special_eos_id
...

[ikawrakow]

The purpose of these <s> and </s> separators is not necessarily to be BOS/EOS tokens. Instead, they serve as separators between the examples and the actual task. My initial attempt to just add 10 examples did not lead to any improvement in the HellaSwag score. I then tried various ways of separating the individual examples. <s> and </s> worked best, with <item>, </item> being close second. If one wanted to add actual BOS/EOS between the examples, one would need to change the hellaswag() implementation to read individual examples and prepare the tokens as needed, instead of just reading lines of text as it is now.

[klosax]

Does the <s></s> also works best in Falcon? It sure looks like the special tags for LLaMA BOS and EOS. Is there any general accepted tags for separating things that should be compatible to any model? Maybe <item></item> should be used?

The hellaswag implementation could parse the input and separate the tokenization and correctly insert BOS or EOS when <s> or </s> is found.

[ikawrakow]

Running 10-shot HellaSwag for Falcon-7b with 5000 tasks on an RTX-4080 takes 1.5 hours. So, I think, I'll let someone else to sort out if there are better example separators for Falcon.

[JianbangZ]

Sorry to bring this up again. Why my hellaswag score so low? for example, both 400 randomized tasks and full 10042 tasks give me 50 ish score using MIstral-7B-v0.1, which is supposed to be in the 80 ish.
I ran the test using fp16 model.
@ikawrakow @klosax

[ikawrakow]

So, if memory serves me correctly, there were changes to the tokenizer since I last used this tool. Again, if I remember correctly, one of the changes was to automatically add a bos token in llama_tokenize. If this memory is correct, this does break the existing HellaSwag implementation.

[JianbangZ]

From the logs, it seems
llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true

[ikawrakow]

Looking at it, there are quite a few changes since my last commit on Aug 26. There were also the KV handling related changes, this could also be causing it.

[ggerganov]

Check if #4981 helps - we forgot to clear the KV cache when we switched to the new API

[ikawrakow]

OK, it is KV cache related. I submitted a PR that fixes it (#4982)