Similar project but faster

[0uep]

Hi Radu,

I have just started a new project BaseXX:
https://github.com/teal-finance/BaseXX

This is very similar to what you did. 😄

I have selected another Base58 project that is faster than the one you choose as original work. You can run the benchmark on your computer, BaseXX is about five times faster:

go test -run=NO -bench=. -benchmem github.com/teal-finance/BaseXX/base92
  
goos: linux
goarch: amd64
pkg: github.com/teal-finance/BaseXX/base92
cpu: AMD Ryzen 9 3900X 12-Core Processor            
BenchmarkEncode-24                       1000000              1944 ns/op              96 B/op          2 allocs/op
BenchmarkEncodeSmartgoBase92-24           135556              8722 ns/op            1377 B/op         78 allocs/op
BenchmarkDecode-24                       1794903               673.5 ns/op           122 B/op          2 allocs/op
BenchmarkDecodeSmartgoBase92-24           339506              3496 ns/op             232 B/op          8 allocs/op
PASS
ok      github.com/teal-finance/BaseXX/base92   6.404s

Have fun 😀

[unix-world]

Hi,
I will look over your project. Looks interesting.

First, every piece of code inside smartgo is threadsafe in multi-threading usage (except webview2 ... which is includded there for another purpose).

Second, the BaseXX project was also ported to javascript and php so my first purpose is to have that logic.
http://demo.unix-world.org/smart-framework/?/page/samples.testunit/tab/0/CamelCase/Test/
Try the "Cryptography Test" area to see what I mean. I was focusing on a portable algorithm between languages...

Thus in multi-threading I was focusing on not having "concurrency issues".
Perhaps yours is the same, but I did not yet look too deep. Have you tested into an async / multi-concurency threaded project in Go ?

[unix-world]

Hi 0uep

Actually you did a possible wrong measure ...
I am not sure how did you get those results :-)
But let's make a simple test.

Simple test: Encode / Decode using a text file (download the following text file, place it in the same folder as the following 2 scripts and you will see...)
https://www.gnu.org/licenses/gpl-3.0.txt

What is more important for you to understand is that this file is only 12kb and the encoding using my version is a bit faster but decoding is 2x faster on my side.
If you test it with 1MB file ... the difference is much much bigger ... you version will be significant slower.
More ... if you use a 64MB text file ... the result of your version is far away slower !

The idea of base92 as the rest is to compress the strings (imagine like archiving).
Using BigInt makes the algorithm portable also to Javascript. Under PHP I implemented it a bit different but the smartgo version of BaseXX algo is thread-safe, solid, stable, very fast including for large strings aka archives.

Just take a look below how results are with a 12KB text:

smartgo/base92 (my project):
ENCODE: 888.378397ms ; DECODE: 191.787475ms

Your project:
ENCODE: 1.082452169s ; DECODE: 254.817341ms

// test-base92-unixworld.go
// unix-world/smartgo/base92 (FASTER) ; ENCODE: 888.378397ms ; DECODE: 191.787475ms

package main

import (
	"log"
	"time"

	smart "github.com/unix-world/smartgo"

	b92 "github.com/unix-world/smartgo/base92"

)

func main() {

	var timerStart time.Time

	txt, _ := smart.SafePathFileRead("gpl-3.0.txt", false) // this is a 12kb text file

	timerStart = time.Now()
	etxt := b92.Encode([]byte(txt))
	durationEnc := time.Since(timerStart)

	timerStart = time.Now()
	dtxt, _ := b92.Decode(etxt)
	durationDec := time.Since(timerStart)

	log.Println(dtxt, etxt, durationEnc, durationDec)

} //END FUNCTION

// test-base92-tealfinance.go 
// teal-finance/BaseXX/base92 (SLOWER) ; ENCODE: 1.082452169s ; DECODE: 254.817341ms

package main

import (
	"log"
	"time"

	smart "github.com/unix-world/smartgo"

	b92 "github.com/teal-finance/BaseXX/base92"

)

func main() {

	var timerStart time.Time

	txt, _ := smart.SafePathFileRead("gpl-3.0.txt", false) // this is a 12kb text file

	timerStart = time.Now()
	etxt := b92.Encode([]byte(txt))
	durationEnc := time.Since(timerStart)

	timerStart = time.Now()
	dtxt, _ := b92.Decode(etxt)
	durationDec := time.Since(timerStart)

	log.Println(dtxt, etxt, durationEnc, durationDec)

} //END FUNCTION

Please notice, I only measured the encoding and decoding time.

As you can see the tests are identical ...

[unix-world]

Just download the https://www.gnu.org/licenses/gpl-3.0.txt and put in the same folder as my tests and see how results are ;-)

[unix-world]

Looking a bit over your code I also noticed some possible multi-concurrency issues which have to be also tested a lot to be sure it does not panic under heavy loads by accessing same areas simultaneous

[unix-world]

Be sure you compile the files using go build if you want a real test. Benchmarking with go as you did measures something different. I measured exactly the encode and decode time not Go benchmark times which are mostly tricky ...

[0uep]

Dear Radu

Thank you for your involvement and your useful replies.
You are right, my implementation is not faster.

I have been inspired by your Go samples and have also coded a bench command:
https://github.com/teal-finance/BaseXX/blob/main/examples/bench/main.go

You can test it right now:

go run github.com/teal-finance/BaseXX/examples/bench -f gpl-3.0.txt

About the multi-concurrency issues, I think read-only data do not need a mutex in Go. You may remove some mutexes in your code…

I will update soon the README of BaseXX to say your project as similar performance with larger samples. ;-)

I think our two projects are too slow compared to https://github.com/mtraver/base91 results. Thus, on the new project https://github.com/teal-finance/incorruptible, I have decided to switch to https://github.com/mtraver/base91 because https://github.com/teal-finance/BaseXX is too slow.

PS: congratulations for your awesome projects, I have been impressed by http://demo.unix-world.org/smart-framework/?/page/samples.testunit/tab/0/CamelCase/Test/

[unix-world]

To be honest, concurrent map read still requires a read mutex in high concurrency.
Read/Write requires a RW Mutex. At least this is what I found out ... based on some experience in real environments.

[0uep]

Thank you for sharing your experience about concurrent map read/write.

I quote your previous message:

Looking a bit over your code I also noticed some possible multi-concurrency issues which have to be also tested a lot to be sure it does not panic under heavy loads by accessing same areas simultaneous

I have not found any mutex in this code:
https://github.com/unix-world/smartgo/blob/master/baseconv/baseconv.go

Neither in standard "encoding/base64":
https://github.com/golang/go/blob/master/src/encoding/base64/base64.go

I think there is no concurrent issue in our both BaseN-related projects.