Merge llama.cpp commit b9fd7ee (remove bit-shuffling) breaking changes to old quantized models

[abetlen]

llama.cpp recently removed bit-shuffling in some of the quantized file formats to improve performance, this a breaking change which currently doesn't have an upgrade path other then re-quantizing the base models (some users only have the quantized ggml models).

I support any change that makes inference faster as I think we still have a long way to go and don't think we should slow down the pace of improvements for what is still essentially still alpha software. That being said I also don't want to cause too much disruption to existing users who have working setups. I'll create a PR for this change and keep it open for at least the weekend before publishing a new PyPI release with the changes. After that the old version will remain available on PyPI and through the releases so people can keep it pinned.

[gjmulder]

I have access to the *.pth files for all of 7B, 13B, 30B and 65B so I can requantize everything if you need any regression testing done. I can't of course share the models because you need to request them from FB. 😜

[snxraven]

Most open and 3rd party models have been updated on Huggingface to support the removal of shuffling, those who use GGML files from there should just need to download the newly updated .bin file. I personally think we should just rip the band-aid and get it over with :P

Those who need to stay with old formats can use: pip install llama-cpp-python==0.1.49 until they update their bins

[TheBloke]

Yeah it'd be great from my point of view if you guys could do the update. I'd like to write instructions telling people how to manually update their text-gen-ui for use with my GGML models.

Today I released my first GGML repo that only works with the updated quant formats.

[gjmulder]

I'm seeing about a 50% performance improvement on 7B q4_1 perplexity testing with a 1080Ti on llama.cpp commit f048af02.

Results to be posted here, as usual.

This will be the last perplexity runs I am doing as next I'm going to try and smoke test the much more appropriate instruction-focused benchmark https://github.com/EleutherAI/lm-evaluation-harness pointed at llama-cpp-python

[TheBloke]

I'm seeing about a 50% performance improvement on 7B q4_1 perplexity testing with a 1080Ti on llama.cpp commit f048af02.

Results to be posted here, as usual.

Very cool. By the way, don't know if this is of interest to you but I recently wrote a Python transformers/pytorch perplexity calculation that 100% replicates llama.cpp's algorithm, returning the exact same score on float16 Llama 7B as llama.cpp does.

On an A6000 GPU it does the full wikitext2 dataset on a float32 Llama 7B in 4 minutes 53 seconds. And in float16 it takes 1 minute 41 seconds.

This compares to llama.cpp + CUBLAS on the same system ETAing 1 hour 13 minutes for float32 Llama 7B and 51 minutes in float16. So that's a 15-30x speed difference.

I've not yet tested it on smaller GPUs or with quantised models, but I will be doing so soon.

Code's not finished yet but the WIP is here if you're interested to take a look at any point: AutoGPTQ/AutoGPTQ#70

[ggerganov]

@TheBloke

Thank you for these results.
Do you have an idea how the A6000 GPU compares to GeForce RTX 4080?

Using the latest llama.cpp I get a F16 perplexity runtime of ~12 minutes with GeForce RTX 4080 and 12 threads on Ryzen 5975WX:

$  make clean && LLAMA_CUBLAS=1 make -j && time ./perplexity -m ./models/7B/ggml-model-f16.bin -f ./build/wiki.test.raw --no-mmap -t 12 -ngl 41

$  make clean && LLAMA_CUBLAS=1 make -j && time ./perplexity -m ./models/7B/ggml-model-f16.bin -f ./build/wiki.test.raw --no-mmap -t 12 -ngl 41
I llama.cpp build info: 
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native
I LDFLAGS:  
I CC:       cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX:      g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

rm -vf *.o main quantize quantize-stats perplexity embedding benchmark-matmult save-load-state build-info.h
removed 'common.o'
removed 'ggml-cuda.o'
removed 'ggml.o'
removed 'llama.o'
removed 'main'
removed 'quantize'
removed 'quantize-stats'
removed 'perplexity'
removed 'embedding'
removed 'build-info.h'
I llama.cpp build info: 
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include
I LDFLAGS:  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
I CC:       cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX:      g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

cc  -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include   -c ggml.c -o ggml.o
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -c llama.cpp -o llama.o
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -c examples/common.cpp -o common.o
nvcc --forward-unknown-to-host-compiler -arch=native -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -Wno-pedantic -c ggml-cuda.cu -o ggml-cuda.o
llama.cpp: In function ‘size_t llama_set_state_data(llama_context*, const uint8_t*)’:
llama.cpp:2684:27: warning: cast from type ‘const uint8_t*’ {aka ‘const unsigned char*’} to type ‘void*’ casts away qualifiers [-Wcast-qual]
 2684 |             kin3d->data = (void *) inp;
      |                           ^~~~~~~~~~~~
llama.cpp:2688:27: warning: cast from type ‘const uint8_t*’ {aka ‘const unsigned char*’} to type ‘void*’ casts away qualifiers [-Wcast-qual]
 2688 |             vin3d->data = (void *) inp;
      |                           ^~~~~~~~~~~~
ggml-cuda.cu(379): warning #177-D: function "convert_mul_mat_vec_f16_cuda" was declared but never referenced
  static void convert_mul_mat_vec_f16_cuda(const void * vx, const float * y, float * dst, const int ncols, const int nrows, cudaStream_t stream) {
              ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/main/main.cpp ggml.o llama.o common.o ggml-cuda.o -o main -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/quantize/quantize.cpp ggml.o llama.o ggml-cuda.o -o quantize -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/quantize-stats/quantize-stats.cpp ggml.o llama.o ggml-cuda.o -o quantize-stats -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/perplexity/perplexity.cpp ggml.o llama.o common.o ggml-cuda.o -o perplexity -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/embedding/embedding.cpp ggml.o llama.o common.o ggml-cuda.o -o embedding -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include pocs/vdot/vdot.cpp ggml.o ggml-cuda.o -o vdot -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib

====  Run ./main -h for help.  ====

main: build = 543 (66841fd)
main: seed  = 1683986883
llama.cpp: loading model from ./models/7B/ggml-model-f16.bin
llama_model_load_internal: format     = ggmf v1 (old version with no mmap support)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 1 (mostly F16)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 13161560.75 KB
llama_model_load_internal: mem required  = 14645.09 MB (+ 1026.00 MB per state)
....................................................................................................
llama_model_load_internal: [cublas] offloading 32 layers to GPU
llama_model_load_internal: [cublas] offloading output layer to GPU
llama_model_load_internal: [cublas] total VRAM used: 12602 MB
llama_init_from_file: kv self size  =  256.00 MB

system_info: n_threads = 12 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
perplexity: calculating perplexity over 655 chunks, batch_size=512
perplexity: 1.07 seconds per pass - ETA 11 minutes
[1]4.2336,[2]4.7097,[3]5.5773,[4]6.1812,[5]6.3001,[6]6.2681,[7]6.4616,[8]6.5542,[9]6.8736,[10]7.1206,[11]7.3177,[12]7.3396,[13]7.2502,[14]7.2970,[15]7.5344,[16]7.1650,[17]7.0580,[18]7.0059,[19]6.6592,[20]6.6463,[21]6.5493,[22]6.3767,[23]6.3378,[24]6.2425,[25]6.2402,[26]6.0788,[27]5.9063,[28]5.8052,[29]5.7164,[30]5.5629,[31]5.5326,[32]5.5547,[33]5.5032,[34]5.5342,[35]5.5575,[36]5.5900,[37]5.5882,[38]5.5984,[39]5.6286,[40]5.6779,[41]5.6867,[42]5.7228,[43]5.6857,[44]5.7408,[45]5.7452,[46]5.7164,[47]5.7391,[48]5.7161,[49]5.7167,[50]5.6771,[51]5.6735,[52]5.6644,[53]5.7098,[54]5.6946,[55]5.6742,[56]5.7028,[57]5.7225,[58]5.7409,[59]5.7584,[60]5.7979,[61]5.7898,[62]5.8468,[63]5.8762,[64]5.8885,[65]5.9300,[66]5.9390,[67]5.9556,[68]5.9701,[69]5.9914,[70]6.0197,[71]6.0398,[72]6.0715,[73]6.1270,[74]6.1315,[75]6.1452,[76]6.1578,[77]6.1680,[78]6.1536,[79]6.1803,[80]6.1746,[81]6.1874,[82]6.1921,[83]6.1434,[84]6.1280,[85]6.1170,[86]6.0963,[87]6.0358,[88]6.0121,[89]5.9930,[90]5.9787,[91]6.0016,[92]5.9954,[93]5.9942,[94]5.9914,[95]6.0186,[96]6.0186,[97]6.0129,[98]6.0081,[99]5.9952,[100]5.9937,[101]6.0170,[102]6.0127,[103]6.0326,[104]6.0389,[105]6.0389,[106]6.0548,[107]6.0539,[108]6.0653,[109]6.0613,[110]6.0585,[111]6.0797,[112]6.1002,[113]6.1012,[114]6.0970,[115]6.1030,[116]6.0941,[117]6.0988,[118]6.1261,[119]6.1472,[120]6.1807,[121]6.1953,[122]6.2198,[123]6.2567,[124]6.2742,[125]6.2662,[126]6.3037,[127]6.3393,[128]6.3686,[129]6.3539,[130]6.3600,[131]6.3560,[132]6.3491,[133]6.3350,[134]6.3442,[135]6.3396,[136]6.3292,[137]6.3214,[138]6.3034,[139]6.2939,[140]6.2906,[141]6.2625,[142]6.2597,[143]6.2301,[144]6.2097,[145]6.2007,[146]6.1901,[147]6.1937,[148]6.1942,[149]6.1892,[150]6.1853,[151]6.1870,[152]6.1776,[153]6.1620,[154]6.1540,[155]6.1593,[156]6.1550,[157]6.1713,[158]6.1757,[159]6.1795,[160]6.1823,[161]6.1948,[162]6.1680,[163]6.1567,[164]6.1342,[165]6.1051,[166]6.0790,[167]6.0438,[168]6.0147,[169]6.0012,[170]5.9914,[171]5.9661,[172]5.9499,[173]5.9340,[174]5.9054,[175]5.8841,[176]5.8726,[177]5.8539,[178]5.8319,[179]5.8160,[180]5.8071,[181]5.7868,[182]5.7697,[183]5.7570,[184]5.7558,[185]5.7489,[186]5.7497,[187]5.7559,[188]5.7524,[189]5.7689,[190]5.7700,[191]5.7907,[192]5.8059,[193]5.8213,[194]5.8321,[195]5.8527,[196]5.8675,[197]5.8883,[198]5.9023,[199]5.9049,[200]5.9100,[201]5.9050,[202]5.9235,[203]5.9304,[204]5.9292,[205]5.9393,[206]5.9459,[207]5.9417,[208]5.9501,[209]5.9541,[210]5.9591,[211]5.9702,[212]5.9772,[213]5.9872,[214]5.9896,[215]5.9933,[216]6.0061,[217]6.0236,[218]6.0364,[219]6.0360,[220]6.0325,[221]6.0268,[222]6.0247,[223]6.0161,[224]6.0096,[225]6.0062,[226]6.0261,[227]6.0337,[228]6.0388,[229]6.0447,[230]6.0418,[231]6.0573,[232]6.0462,[233]6.0298,[234]6.0158,[235]5.9969,[236]5.9910,[237]5.9818,[238]5.9840,[239]5.9700,[240]5.9604,[241]5.9619,[242]5.9651,[243]5.9637,[244]5.9530,[245]5.9495,[246]5.9392,[247]5.9277,[248]5.9206,[249]5.9176,[250]5.9220,[251]5.9154,[252]5.9121,[253]5.9026,[254]5.8973,[255]5.8865,[256]5.8697,[257]5.8576,[258]5.8495,[259]5.8468,[260]5.8385,[261]5.8340,[262]5.8289,[263]5.8232,[264]5.8023,[265]5.8018,[266]5.7997,[267]5.7937,[268]5.8023,[269]5.8004,[270]5.8017,[271]5.8093,[272]5.8126,[273]5.8133,[274]5.8152,[275]5.8233,[276]5.8293,[277]5.8444,[278]5.8540,[279]5.8632,[280]5.8662,[281]5.8758,[282]5.8820,[283]5.8962,[284]5.9038,[285]5.9121,[286]5.9250,[287]5.9242,[288]5.9296,[289]5.9219,[290]5.9068,[291]5.8926,[292]5.8783,[293]5.8657,[294]5.8677,[295]5.8667,[296]5.8713,[297]5.8703,[298]5.8732,[299]5.8709,[300]5.8607,[301]5.8608,[302]5.8534,[303]5.8450,[304]5.8369,[305]5.8334,[306]5.8213,[307]5.8236,[308]5.8261,[309]5.8111,[310]5.8060,[311]5.8000,[312]5.8022,[313]5.7967,[314]5.7951,[315]5.7800,[316]5.7752,[317]5.7598,[318]5.7405,[319]5.7519,[320]5.7637,[321]5.7683,[322]5.7642,[323]5.7578,[324]5.7551,[325]5.7651,[326]5.7655,[327]5.7672,[328]5.7707,[329]5.7758,[330]5.7783,[331]5.7903,[332]5.7878,[333]5.7943,[334]5.7891,[335]5.7835,[336]5.7870,[337]5.7851,[338]5.7846,[339]5.7796,[340]5.7758,[341]5.7829,[342]5.7857,[343]5.7905,[344]5.7909,[345]5.7915,[346]5.7891,[347]5.7933,[348]5.7967,[349]5.7993,[350]5.7965,[351]5.7975,[352]5.7972,[353]5.7918,[354]5.7919,[355]5.7969,[356]5.8002,[357]5.7967,[358]5.8056,[359]5.8082,[360]5.8052,[361]5.8047,[362]5.8116,[363]5.8225,[364]5.8282,[365]5.8328,[366]5.8339,[367]5.8420,[368]5.8396,[369]5.8407,[370]5.8423,[371]5.8378,[372]5.8427,[373]5.8471,[374]5.8457,[375]5.8459,[376]5.8525,[377]5.8491,[378]5.8519,[379]5.8575,[380]5.8500,[381]5.8469,[382]5.8420,[383]5.8410,[384]5.8405,[385]5.8395,[386]5.8390,[387]5.8389,[388]5.8356,[389]5.8307,[390]5.8240,[391]5.8167,[392]5.8129,[393]5.8113,[394]5.8140,[395]5.8129,[396]5.8060,[397]5.8127,[398]5.8163,[399]5.8238,[400]5.8237,[401]5.8251,[402]5.8262,[403]5.8283,[404]5.8345,[405]5.8251,[406]5.8218,[407]5.8213,[408]5.8229,[409]5.8342,[410]5.8449,[411]5.8559,[412]5.8715,[413]5.8821,[414]5.8897,[415]5.8951,[416]5.9027,[417]5.9143,[418]5.9180,[419]5.9242,[420]5.9326,[421]5.9438,[422]5.9477,[423]5.9543,[424]5.9647,[425]5.9730,[426]5.9793,[427]5.9836,[428]5.9920,[429]5.9966,[430]6.0047,[431]6.0180,[432]6.0226,[433]6.0219,[434]6.0176,[435]6.0188,[436]6.0212,[437]6.0304,[438]6.0379,[439]6.0353,[440]6.0344,[441]6.0297,[442]6.0283,[443]6.0296,[444]6.0301,[445]6.0281,[446]6.0305,[447]6.0331,[448]6.0371,[449]6.0348,[450]6.0357,[451]6.0320,[452]6.0190,[453]6.0106,[454]6.0053,[455]6.0063,[456]6.0108,[457]6.0128,[458]6.0107,[459]6.0110,[460]6.0194,[461]6.0169,[462]6.0155,[463]6.0190,[464]6.0179,[465]6.0153,[466]6.0078,[467]6.0080,[468]6.0078,[469]6.0098,[470]6.0102,[471]6.0056,[472]6.0098,[473]6.0048,[474]6.0061,[475]6.0003,[476]6.0016,[477]5.9947,[478]5.9936,[479]5.9992,[480]6.0035,[481]6.0052,[482]6.0010,[483]5.9971,[484]5.9991,[485]5.9969,[486]5.9915,[487]5.9913,[488]5.9888,[489]5.9842,[490]5.9820,[491]5.9793,[492]5.9740,[493]5.9716,[494]5.9702,[495]5.9697,[496]5.9660,[497]5.9607,[498]5.9590,[499]5.9550,[500]5.9461,[501]5.9399,[502]5.9401,[503]5.9394,[504]5.9309,[505]5.9330,[506]5.9339,[507]5.9279,[508]5.9237,[509]5.9232,[510]5.9261,[511]5.9307,[512]5.9341,[513]5.9362,[514]5.9424,[515]5.9371,[516]5.9361,[517]5.9372,[518]5.9368,[519]5.9399,[520]5.9422,[521]5.9433,[522]5.9459,[523]5.9466,[524]5.9523,[525]5.9552,[526]5.9559,[527]5.9577,[528]5.9527,[529]5.9531,[530]5.9482,[531]5.9472,[532]5.9518,[533]5.9542,[534]5.9526,[535]5.9547,[536]5.9496,[537]5.9475,[538]5.9524,[539]5.9535,[540]5.9572,[541]5.9572,[542]5.9585,[543]5.9601,[544]5.9611,[545]5.9591,[546]5.9599,[547]5.9560,[548]5.9516,[549]5.9518,[550]5.9491,[551]5.9459,[552]5.9438,[553]5.9403,[554]5.9384,[555]5.9355,[556]5.9350,[557]5.9374,[558]5.9338,[559]5.9332,[560]5.9330,[561]5.9333,[562]5.9312,[563]5.9308,[564]5.9349,[565]5.9367,[566]5.9366,[567]5.9346,[568]5.9353,[569]5.9341,[570]5.9368,[571]5.9373,[572]5.9384,[573]5.9383,[574]5.9347,[575]5.9338,[576]5.9336,[577]5.9321,[578]5.9303,[579]5.9306,[580]5.9241,[581]5.9207,[582]5.9197,[583]5.9206,[584]5.9209,[585]5.9135,[586]5.9069,[587]5.9075,[588]5.9124,[589]5.9175,[590]5.9203,[591]5.9223,[592]5.9211,[593]5.9181,[594]5.9193,[595]5.9170,[596]5.9203,[597]5.9184,[598]5.9157,[599]5.9178,[600]5.9175,[601]5.9160,[602]5.9170,[603]5.9197,[604]5.9205,[605]5.9239,[606]5.9258,[607]5.9243,[608]5.9210,[609]5.9219,[610]5.9251,[611]5.9233,[612]5.9260,[613]5.9225,[614]5.9179,[615]5.9110,[616]5.9134,[617]5.9074,[618]5.9026,[619]5.8974,[620]5.8844,[621]5.8781,[622]5.8766,[623]5.8782,[624]5.8788,[625]5.8790,[626]5.8780,[627]5.8803,[628]5.8804,[629]5.8800,[630]5.8831,[631]5.8883,[632]5.8937,[633]5.8924,[634]5.8958,[635]5.8967,[636]5.8934,[637]5.8900,[638]5.8923,[639]5.8892,[640]5.8902,[641]5.8904,[642]5.8969,[643]5.8989,[644]5.9001,[645]5.8984,[646]5.9024,[647]5.8986,[648]5.8995,[649]5.8997,[650]5.9033,[651]5.9083,[652]5.9094,[653]5.9133,[654]5.9072,[655]5.9066,

llama_print_timings:        load time =  7138.61 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per token)
llama_print_timings: prompt eval time = 690434.51 ms / 335360 tokens (    2.06 ms per token)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token)
llama_print_timings:       total time = 730450.98 ms

real	12m11,751s
user	137m8,912s
sys	0m18,541s

[gjmulder]

@ggerganov @TheBloke in terms of memory bandwidth they're roughly similar. Core clock speeds look to be about 50% faster for the RTX 4080.

[TheBloke]

Apologies @ggerganov - my earlier comparison on llama.cpp times was incorrect and misleading, because I wasn't using the right command line arguments. So it wasn't doing the cublas offloading.

I ran your command exactly, and the time on the A6000 + 12 threads on AMD EPYC 7352 24-Core Processor, for huggyllama_7b in fp16 GGML was: 18 minutes 44 seconds.

So the pytorch ppl code is faster, but not such a vast difference as I wrongly thought earlier.

I've not run the pytorch ppl on a 4080 yet but based on Gary's message I guess it will be slightly faster, maybe 1 minute 20 or around there for 7B float16.

llama.cpp quantize log on A6000 + 12 threads of AMD EPYC 7352 24-core processor

root@4932db78a9e1:~/llama.cpp# make clean && LLAMA_CUBLAS=1 make -j && time ./perplexity -m /workspace/huggy7b.fp16.bin -f /workspace/wikitext-2-raw/wiki.test.raw --no-mmap -t 12 -ngl 41
I llama.cpp build info:
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native
I LDFLAGS:
I CC:       cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
I CXX:      g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

rm -vf *.o main quantize quantize-stats perplexity embedding benchmark-matmult save-load-state build-info.h
I llama.cpp build info:
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include
I LDFLAGS:   -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
I CC:       cc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
I CXX:      g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

cc  -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include   -c ggml.c -o ggml.o
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -c llama.cpp -o llama.o
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -c examples/common.cpp -o common.o
nvcc --forward-unknown-to-host-compiler -arch=native -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -Wno-pedantic -c ggml-cuda.cu -o ggml-cuda.o
llama.cpp: In function ‘size_t llama_set_state_data(llama_context*, const uint8_t*)’:
llama.cpp:2686:36: warning: cast from type ‘const uint8_t*’ {aka ‘const unsigned char*’} to type ‘void*’ casts away qualifiers [-Wcast-qual]
 2686 |             kin3d->data = (void *) inp;
      |                                    ^~~
llama.cpp:2690:36: warning: cast from type ‘const uint8_t*’ {aka ‘const unsigned char*’} to type ‘void*’ casts away qualifiers [-Wcast-qual]
 2690 |             vin3d->data = (void *) inp;
      |                                    ^~~
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/main/main.cpp ggml.o llama.o common.o ggml-cuda.o -o main  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/quantize/quantize.cpp ggml.o llama.o ggml-cuda.o -o quantize  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/quantize-stats/quantize-stats.cpp ggml.o llama.o ggml-cuda.o -o quantize-stats  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/perplexity/perplexity.cpp ggml.o llama.o common.o ggml-cuda.o -o perplexity  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include examples/embedding/embedding.cpp ggml.o llama.o common.o ggml-cuda.o -o embedding  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib
g++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include pocs/vdot/vdot.cpp ggml.o ggml-cuda.o -o vdot  -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib

====  Run ./main -h for help.  ====

main: build = 546 (08737ef)
main: seed  = 1683990420
llama.cpp: loading model from /workspace/huggy7b.fp16.bin
llama_model_load_internal: format     = ggjt v1 (pre #1405)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 1 (mostly F16)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 13161560.75 KB
llama_model_load_internal: mem required  = 14645.09 MB (+ 1026.00 MB per state)
....................................................................................................
llama_model_load_internal: [cublas] offloading 32 layers to GPU
llama_model_load_internal: [cublas] offloading output layer to GPU
llama_model_load_internal: [cublas] total VRAM used: 12602 MB
llama_init_from_file: kv self size  =  256.00 MB

system_info: n_threads = 12 / 48 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
perplexity: calculating perplexity over 655 chunks, batch_size=512
perplexity: 1.50 seconds per pass - ETA 16 minutes
[1]4.2336,[2]4.7097,[3]5.5773,[4]6.1812,[5]6.3001,[6]6.2681,[7]6.4616,[8]6.5542,[9]6.8736,[10]7.1206,[11]7.3177,[12]7.3396,[13]7.2502,[14]7.2970,[15]7.5344,[16]7.1650,[17]7.0580,[18]7.0059,[19]6.6592,[20]6.6463,[21]6.5493,[22]6.3767,[23]6.3378,[24]6.2425,[25]6.2402,[26]6.0788,[27]5.9063,[28]5.8052,[29]5.7164,[30]5.5629,[31]5.5326,[32]5.5547,[33]5.5032,[34]5.5342,[35]5.5575,[36]5.5900,[37]5.5882,[38]5.5984,[39]5.6286,[40]5.6779,[41]5.6867,[42]5.7228,[43]5.6857,[44]5.7408,[45]5.7452,[46]5.7164,[47]5.7391,[48]5.7161,[49]5.7167,[50]5.6771,[51]5.6735,[52]5.6644,[53]5.7098,[54]5.6946,[55]5.6742,[56]5.7028,[57]5.7225,[58]5.7409,[59]5.7584,[60]5.7979,[61]5.7898,[62]5.8468,[63]5.8762,[64]5.8885,[65]5.9300,[66]5.9390,[67]5.9556,[68]5.9701,[69]5.9914,[70]6.0197,[71]6.0398,[72]6.0715,[73]6.1270,[74]6.1315,[75]6.1452,[76]6.1578,[77]6.1680,[78]6.1536,[79]6.1803,[80]6.1746,[81]6.1874,[82]6.1921,[83]6.1434,[84]6.1280,[85]6.1170,[86]6.0963,[87]6.0358,[88]6.0121,[89]5.9930,[90]5.9787,[91]6.0016,[92]5.9954,[93]5.9942,[94]5.9914,[95]6.0186,[96]6.0186,[97]6.0129,[98]6.0081,[99]5.9952,[100]5.9937,[101]6.0170,[102]6.0127,[103]6.0326,[104]6.0389,[105]6.0389,[106]6.0548,[107]6.0539,[108]6.0653,[109]6.0613,[110]6.0585,[111]6.0797,[112]6.1002,[113]6.1012,[114]6.0970,[115]6.1030,[116]6.0941,[117]6.0988,[118]6.1261,[119]6.1472,[120]6.1807,[121]6.1953,[122]6.2198,[123]6.2567,[124]6.2742,[125]6.2662,[126]6.3037,[127]6.3393,[128]6.3686,[129]6.3539,[130]6.3600,[131]6.3560,[132]6.3491,[133]6.3350,[134]6.3442,[135]6.3396,[136]6.3292,[137]6.3214,[138]6.3034,[139]6.2939,[140]6.2906,[141]6.2625,[142]6.2597,[143]6.2301,[144]6.2097,[145]6.2007,[146]6.1901,[147]6.1937,[148]6.1942,[149]6.1892,[150]6.1853,[151]6.1870,[152]6.1776,[153]6.1620,[154]6.1540,[155]6.1593,[156]6.1550,[157]6.1713,[158]6.1757,[159]6.1795,[160]6.1823,[161]6.1948,[162]6.1680,[163]6.1567,[164]6.1342,[165]6.1051,[166]6.0790,[167]6.0438,[168]6.0147,[169]6.0012,[170]5.9914,[171]5.9661,[172]5.9499,[173]5.9340,[174]5.9054,[175]5.8841,[176]5.8726,[177]5.8539,[178]5.8319,[179]5.8160,[180]5.8071,[181]5.7868,[182]5.7697,[183]5.7570,[184]5.7558,[185]5.7489,[186]5.7497,[187]5.7559,[188]5.7524,[189]5.7689,[190]5.7700,[191]5.7907,[192]5.8059,[193]5.8213,[194]5.8321,[195]5.8527,[196]5.8675,[197]5.8883,[198]5.9023,[199]5.9049,[200]5.9100,[201]5.9050,[202]5.9235,[203]5.9304,[204]5.9292,[205]5.9393,[206]5.9459,[207]5.9417,[208]5.9501,[209]5.9541,[210]5.9591,[211]5.9702,[212]5.9772,[213]5.9872,[214]5.9896,[215]5.9933,[216]6.0061,[217]6.0236,[218]6.0364,[219]6.0360,[220]6.0325,[221]6.0268,[222]6.0247,[223]6.0161,[224]6.0096,[225]6.0062,[226]6.0261,[227]6.0337,[228]6.0388,[229]6.0447,[230]6.0418,[231]6.0573,[232]6.0462,[233]6.0298,[234]6.0158,[235]5.9969,[236]5.9910,[237]5.9818,[238]5.9840,[239]5.9700,[240]5.9604,[241]5.9619,[242]5.9651,[243]5.9637,[244]5.9530,[245]5.9495,[246]5.9392,[247]5.9277,[248]5.9206,[249]5.9176,[250]5.9220,[251]5.9154,[252]5.9121,[253]5.9026,[254]5.8973,[255]5.8865,[256]5.8697,[257]5.8576,[258]5.8495,[259]5.8468,[260]5.8385,[261]5.8340,[262]5.8289,[263]5.8232,[264]5.8023,[265]5.8018,[266]5.7997,[267]5.7937,[268]5.8023,[269]5.8004,[270]5.8017,[271]5.8093,[272]5.8126,[273]5.8133,[274]5.8152,[275]5.8233,[276]5.8293,[277]5.8444,[278]5.8540,[279]5.8632,[280]5.8662,[281]5.8758,[282]5.8820,[283]5.8962,[284]5.9038,[285]5.9121,[286]5.9250,[287]5.9242,[288]5.9296,[289]5.9219,[290]5.9068,[291]5.8926,[292]5.8783,[293]5.8657,[294]5.8677,[295]5.8667,[296]5.8713,[297]5.8703,[298]5.8732,[299]5.8709,[300]5.8607,[301]5.8608,[302]5.8534,[303]5.8450,[304]5.8369,[305]5.8334,[306]5.8213,[307]5.8236,[308]5.8261,[309]5.8111,[310]5.8060,[311]5.8000,[312]5.8022,[313]5.7967,[314]5.7951,[315]5.7800,[316]5.7752,[317]5.7598,[318]5.7405,[319]5.7519,[320]5.7637,[321]5.7683,[322]5.7642,[323]5.7578,[324]5.7551,[325]5.7651,[326]5.7655,[327]5.7672,[328]5.7707,[329]5.7758,[330]5.7783,[331]5.7903,[332]5.7878,[333]5.7943,[334]5.7891,[335]5.7835,[336]5.7870,[337]5.7851,[338]5.7846,[339]5.7796,[340]5.7758,[341]5.7829,[342]5.7857,[343]5.7905,[344]5.7909,[345]5.7915,[346]5.7891,[347]5.7933,[348]5.7967,[349]5.7993,[350]5.7965,[351]5.7975,[352]5.7972,[353]5.7918,[354]5.7919,[355]5.7969,[356]5.8002,[357]5.7967,[358]5.8056,[359]5.8082,[360]5.8052,[361]5.8047,[362]5.8116,[363]5.8225,[364]5.8282,[365]5.8328,[366]5.8339,[367]5.8420,[368]5.8396,[369]5.8407,[370]5.8423,[371]5.8378,[372]5.8427,[373]5.8471,[374]5.8457,[375]5.8459,[376]5.8525,[377]5.8491,[378]5.8519,[379]5.8575,[380]5.8500,[381]5.8469,[382]5.8420,[383]5.8410,[384]5.8405,[385]5.8395,[386]5.8390,[387]5.8389,[388]5.8356,[389]5.8307,[390]5.8240,[391]5.8167,[392]5.8129,[393]5.8113,[394]5.8140,[395]5.8129,[396]5.8060,[397]5.8127,[398]5.8163,[399]5.8238,[400]5.8237,[401]5.8251,[402]5.8262,[403]5.8283,[404]5.8345,[405]5.8251,[406]5.8218,[407]5.8213,[408]5.8229,[409]5.8342,[410]5.8449,[411]5.8559,[412]5.8715,[413]5.8821,[414]5.8897,[415]5.8951,[416]5.9027,[417]5.9143,[418]5.9180,[419]5.9242,[420]5.9326,[421]5.9438,[422]5.9477,[423]5.9543,[424]5.9647,[425]5.9730,[426]5.9793,[427]5.9836,[428]5.9920,[429]5.9966,[430]6.0047,[431]6.0180,[432]6.0226,[433]6.0219,[434]6.0176,[435]6.0188,[436]6.0212,[437]6.0304,[438]6.0379,[439]6.0353,[440]6.0344,[441]6.0297,[442]6.0283,[443]6.0296,[444]6.0301,[445]6.0281,[446]6.0305,[447]6.0331,[448]6.0371,[449]6.0348,[450]6.0357,[451]6.0320,[452]6.0190,[453]6.0106,[454]6.0053,[455]6.0063,[456]6.0108,[457]6.0128,[458]6.0107,[459]6.0110,[460]6.0194,[461]6.0169,[462]6.0155,[463]6.0190,[464]6.0179,[465]6.0153,[466]6.0078,[467]6.0080,[468]6.0078,[469]6.0098,[470]6.0102,[471]6.0056,[472]6.0098,[473]6.0048,[474]6.0061,[475]6.0003,[476]6.0016,[477]5.9947,[478]5.9936,[479]5.9992,[480]6.0035,[481]6.0052,[482]6.0010,[483]5.9971,[484]5.9991,[485]5.9969,[486]5.9915,[487]5.9913,[488]5.9888,[489]5.9842,[490]5.9820,[491]5.9793,[492]5.9740,[493]5.9716,[494]5.9702,[495]5.9697,[496]5.9660,[497]5.9607,[498]5.9590,[499]5.9550,[500]5.9461,[501]5.9399,[502]5.9401,[503]5.9394,[504]5.9309,[505]5.9330,[506]5.9339,[507]5.9279,[508]5.9237,[509]5.9232,[510]5.9261,[511]5.9307,[512]5.9341,[513]5.9362,[514]5.9424,[515]5.9371,[516]5.9361,[517]5.9372,[518]5.9368,[519]5.9399,[520]5.9422,[521]5.9433,[522]5.9459,[523]5.9466,[524]5.9523,[525]5.9552,[526]5.9559,[527]5.9577,[528]5.9527,[529]5.9531,[530]5.9482,[531]5.9472,[532]5.9518,[533]5.9542,[534]5.9526,[535]5.9547,[536]5.9496,[537]5.9475,[538]5.9524,[539]5.9535,[540]5.9572,[541]5.9572,[542]5.9585,[543]5.9601,[544]5.9611,[545]5.9591,[546]5.9599,[547]5.9560,[548]5.9516,[549]5.9518,[550]5.9491,[551]5.9459,[552]5.9438,[553]5.9403,[554]5.9384,[555]5.9355,[556]5.9350,[557]5.9374,[558]5.9338,[559]5.9332,[560]5.9330,[561]5.9333,[562]5.9312,[563]5.9308,[564]5.9349,[565]5.9367,[566]5.9366,[567]5.9346,[568]5.9353,[569]5.9341,[570]5.9368,[571]5.9373,[572]5.9384,[573]5.9383,[574]5.9347,[575]5.9338,[576]5.9336,[577]5.9321,[578]5.9303,[579]5.9306,[580]5.9241,[581]5.9207,[582]5.9197,[583]5.9206,[584]5.9209,[585]5.9135,[586]5.9069,[587]5.9075,[588]5.9124,[589]5.9175,[590]5.9203,[591]5.9223,[592]5.9211,[593]5.9181,[594]5.9193,[595]5.9170,[596]5.9203,[597]5.9184,[598]5.9157,[599]5.9178,[600]5.9175,[601]5.9160,[602]5.9170,[603]5.9197,[604]5.9205,[605]5.9239,[606]5.9258,[607]5.9243,[608]5.9210,[609]5.9219,[610]5.9251,[611]5.9233,[612]5.9260,[613]5.9225,[614]5.9179,[615]5.9110,[616]5.9134,[617]5.9074,[618]5.9026,[619]5.8974,[620]5.8844,[621]5.8781,[622]5.8766,[623]5.8782,[624]5.8788,[625]5.8790,[626]5.8780,[627]5.8803,[628]5.8804,[629]5.8800,[630]5.8831,[631]5.8883,[632]5.8937,[633]5.8924,[634]5.8958,[635]5.8967,[636]5.8934,[637]5.8900,[638]5.8923,[639]5.8892,[640]5.8902,[641]5.8904,[642]5.8969,[643]5.8989,[644]5.9001,[645]5.8984,[646]5.9024,[647]5.8986,[648]5.8995,[649]5.8997,[650]5.9033,[651]5.9083,[652]5.9094,[653]5.9133,[654]5.9072,[655]5.9066,

llama_print_timings:        load time = 29867.57 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per token)
llama_print_timings: prompt eval time = 1044191.22 ms / 335360 tokens (    3.11 ms per token)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token)
llama_print_timings:       total time = 1120580.04 ms

real	18m44.130s
user	207m10.864s
sys	0m44.931s
root@4932db78a9e1:~/llama.cpp#