Code Challenge Solutions-Pranshu Lakhotia

[PranshuLakhotia]

I was already intrigued by the XLNS project and I decided to take on the code challenges for XLNScpp as well. It was very fun solving the challenges and I gained more experience from it.
Sharing with you my solutions for your valuable feedbacks and suggestions @markgarnold @pradeeban sir.

[PranshuLakhotia]

Code Challenge 1:
Run the xlns16test.cpp and xlns32test.cpp examples

It was very straightforward and simple.

[PranshuLakhotia]

Code Challenge 3:
Write a standalone C++ program that has a function to do 32-bit FP matrix multiply.

The nested loops iterate over rows of the first matrix and columns of the transposed second matrix to calculate each element of the resultant matrix as a sum of products.
Took the same matrix example as it was given on ggml example.
Went through the article of hugging face which involved ggml_backend which I found really interesting(I will spend more time to completely understand it).
I felt that implementation part was fairly simple.

[PranshuLakhotia]

Code Challenge 4:
Modify this program to include xlns32.cpp (define xlns_ideal first) and perform the internal computation in LNS format.
My code takes advantage of the overloaded operators in the xlns32_float class (aliased as xlns) as also given in the hint:
xlns a_lns = A[i][k] : This uses the overloaded assignment operator to convert from float to LNS
result[i][j] = sum : This uses the xlns32_2float function to convert from LNS back to float

Used LNS for internal computation.

To verify that the LNS results are close to the FP results I compared the results element by element as shown in the output. The extra effort was worth it!

[PranshuLakhotia]

@markgarnold sir Updated code and Output with computing the relative difference:

[PranshuLakhotia]

Will be sharing code challenge 5 in a while sir.
Thank you

[markgarnold]

Hi @PranshuLakhotia ,

I definitely encourage you to consider submitting a proposal for this xlnscpp project. It is good you have knowledge of both the xlns and xlnscpp project as the latter is a more efficient implementation of a subset of what xlns provides. There are some python scripts the valildate the addition operations of xlnscpp are identical to that of xlns for a subset of values. Unlike xlns, xlnscpp values can underflow and overflow, which make this package more realistic.

As to your code challenge #3, I would suggest computing the relative difference:

diff = fabs(resultFP[i] - result[i])/resultFP[i];

because both FP and LNS work with relative precision not absolute precision.

[markgarnold]

I meant code challenge #4

[PranshuLakhotia]

I meant code challenge #4

Got it @markgarnold sir will share with you the updated code thank you for the review. Learning a lot from these discussions.
Thank you for the encouragement sir! I'm definitely interested in submitting a proposal for the xlnscpp project. (I am thinking to submit proposals in both of the projects frankly sir). Will go through the sixth research paper as well which you have uploaded for this project(previous 5 are the same only of xlns) for better insights.

[PranshuLakhotia]

Code Challenge 5:
Modify the program to include xlns16.cpp instead.

In this the code remains the same as in xlns32.cpp we just replace xlns32 with xlns16.
The main and important point to notice is that the 16-bit LNS results are less accurate than the 32-bit LNS results from Code Challenge 4 as shown in the Output. This demonstrates the tradeoff between precision and efficiency.

This is because:
It has less precision in the mantissa portion of the logarithmic representation.
16-bit LNS has a smaller range of representable values.
The approximations in the addition operations have more significant rounding errors.

This shows how reduced precision LNS affects numerical accuracy which is a key consideration when implementing LNS in hardware for applications like LLMs where the tradeoff between accuracy and efficiency is important.

[markgarnold]

Hi @PranshuLakhotia you say "16-bit LNS has a smaller range of representable values." That is false. The range is the same (about 1E-38 to 1E37). Only the precision is lower.

[PranshuLakhotia]

Hi @PranshuLakhotia you say "16-bit LNS has a smaller range of representable values." That is false. The range is the same (about 1E-38 to 1E37). Only the precision is lower.

Hi @markgarnold Yes sir I made a mistake I comprehended it now as you pointed out. I incorrectly conflated range with precision. In LNS unlike standard integer representations where more bits directly means larger representable numbers the logarithmic nature means that even with fewer bits we can represent very large and very small numbers just with less precision. Thank you sir.

[PranshuLakhotia]

@pradeeban @markgarnold Sir completed all the Code Challenges. Please review Thank you.

[pradeeban]

Impressive work, @PranshuLakhotia. Thanks! (waiting for @markgarnold for specific comments)

[PranshuLakhotia]

Thanks a lot @pradeeban sir.