Add support for multiple old:new replace mappings

[carterpeel]

Hello,

I'm curious to know if it'd be possible to implement the ability for ios.NewBytesReplacingReader to ingest multiple old:new token mappings in the ios package.

This is obviously the current functionality of the interface:

package main
import (
    "io"
    "os"
    "strings"
    "github.com/jf-tech/go-corelib/ios"
)
func main() {
    r := strings.NewReader("foo bar baz")
    replacer := ios.NewBytesReplacingReader(r, []byte("foo bar baz"), []byte("baz bar foo"))
    io.Copy(os.Stdout, replacer)
}

Would it be too much trouble to get the functionality shown below?

package main
import (
    "io"
    "os"
    "strings"
    "github.com/jf-tech/go-corelib/ios"
)
func main() {
  r := strings.NewReader("foo bar baz")

  replacer := ios.NewBytesReplacingReader(r, [][]byte{[]byte("foo"), []byte("bar"), []byte("baz")}, [][]byte{[]byte("foo-new"), []byte("bar-new"), []byte("baz-new")})

  io.Copy(os.Stdout, replacer) // returns "foo-new bar-new baz-new" to stdout
}

Thanks a bunch.

[jf-tech]

1. Needs more clarification on what the desired behavior should be, if search1 is a prefix of search2 or vice versa.

2. If the answer for 1 is strictly based on the order of search strings provided, then you can simply achieve that goal by wrapping/nesting the replacer:

   replacer := ios.NewBytesReplacingReader(
       ios.NewBytesReplacingReader(
           ios.NewBytesReplacingReader(r, []byte("foo"), []byte("foo-new")),
           []byte("bar"), []byte("bar-new")),
       []byte("baz"), []byte("baz-new"))
   

   the api is written with perf very much among the primary considerations, this wrapping/nesting would be almost
   equally fast as your proposed approach.

   Now if you say the nesting usage looks ugly, I wouldn't mind adding a new API (say NewMultiBytesReplacingReader) which simply takes a slice of []byte for search and slice of []byte for replace and inside does the nesting for user.

[carterpeel]

Apologies for the lack of clarification - I should have been more specific.

I would greatly appreciate it if you could add the NewMultiBytesReplacingReader API method. What I'm writing already has two [][]byte slices, ([]old:[]new) so it would be awesome if the API had the functionality to ingest them by default.

Thanks a ton for the quick response!

[carterpeel]

1. Needs more clarification on what the desired behavior should be, if search1 is a prefix of search2 or vice versa.

2. If the answer for 1 is strictly based on the order of search strings provided, then you can simply achieve that goal by wrapping/nesting the replacer:

   replacer := ios.NewBytesReplacingReader(
       ios.NewBytesReplacingReader(
           ios.NewBytesReplacingReader(r, []byte("foo"), []byte("foo-new")),
           []byte("bar"), []byte("bar-new")),
       []byte("baz"), []byte("baz-new"))
   

   the api is written with perf very much among the primary considerations, this wrapping/nesting would be almost
   equally fast as your proposed approach.
   Now if you say the nesting usage looks ugly, I wouldn't mind adding a new API (say NewMultiBytesReplacingReader) which simply takes a slice of []byte for search and slice of []byte for replace and inside does the nesting for user.

Also, quick question - wouldn't the nested approach example you provided be extremely memory intensive if there are a large number of old:new tokens? From what I understand, ios.NewBytesReplacingReader allocates a new buffer for each new reader. Wouldn't it be more efficient to iterate over the old:new mappings on a single buffer?

[jf-tech]

1. Needs more clarification on what the desired behavior should be, if search1 is a prefix of search2 or vice versa.

2. If the answer for 1 is strictly based on the order of search strings provided, then you can simply achieve that goal by wrapping/nesting the replacer:

   replacer := ios.NewBytesReplacingReader(
       ios.NewBytesReplacingReader(
           ios.NewBytesReplacingReader(r, []byte("foo"), []byte("foo-new")),
           []byte("bar"), []byte("bar-new")),
       []byte("baz"), []byte("baz-new"))
   

   the api is written with perf very much among the primary considerations, this wrapping/nesting would be almost
   equally fast as your proposed approach.
   Now if you say the nesting usage looks ugly, I wouldn't mind adding a new API (say NewMultiBytesReplacingReader) which simply takes a slice of []byte for search and slice of []byte for replace and inside does the nesting for user.

Also, quick question - wouldn't the nested approach example you provided be extremely memory intensive if there are a large number of old:new tokens? From what I understand, ios.NewBytesReplacingReader allocates a new buffer for each new reader. Wouldn't it be more efficient to iterate over the old:new mappings on a single buffer?

That was why I was asking your scenarios and usage patterns. So your use case involve large number of old/new pairs? Large is how large. Second question, is the stream big? Do you have to use this API in streaming fashion or if you can use the replacer to replace old1 with new1, reset the replacer, and then go through the input again to replace old2 with new2, etc?

[carterpeel]

Question 1:
Yes, I'll have large numbers of old:new token pairs.
I also expect to have large old:new token pair sizes (hard to put a number on this, but potentially >500MiB. I've tested it with sizes of over 1GiB and it works fine.)

Question 2:
Regarding the stream size, the stream may be large or small. It is of an arbitrary size. It could be just a couple KiB, or it could be 1TiB.
And yes - it has to be used in a streaming fashion, else, it would require horrendously excessive disk operations to be involved. (either that or store the entire thing in RAM, which isn't an option)

Hopefully that's enough clarification? Apologies if not.

[jf-tech]

fyi, @carterpeel v0.0.15 has been released with the change. let me know if you encounter any issues.