Request for Compatibility with Open-Source Search Engines and JSON Processing Libraries

[totorofly]

Did you search GitHub Issues and GitHub Discussions First?
Yes

Is your feature request related to a problem? Please describe.

I am considering transitioning to your impressive Dragonfly data store for my application due to the performance benefits and vertical scalability it offers.

My application involves a workload that retrieves 2000-3000 keys per second, each with a short lifespan of around 8 minutes. With my current setup using Redis, the write pressure has been significant and has affected the performance of other functionalities like search and JSON processing, which are implemented using RediSearch and RediJSON respectively.

Describe the solution you'd like

Given the high throughput and efficient memory usage of Dragonfly, I believe it could be a better fit for my application. However, I also need to maintain the search and JSON processing functionalities that I currently have with RediSearch and RediJSON.

As Dragonfly is fully compatible with Redis at the API level, theoretically any search engine or JSON processing library that can use Redis as a data source should be able to use Dragonfly as well. However, I'm unable to find concrete information on this, and testing with open-source search engines like Elasticsearch, Apache Solr, Apache Lucene or JSON processing libraries would require a significant investment of time and resources.

Describe alternatives you've considered

Hence, I am writing to kindly request if you could consider providing compatibility with open-source search engines and JSON processing libraries, or recommending some that you know work well with Dragonfly. This would greatly help users like me who require powerful search and JSON processing functionalities in addition to the high-performance data storage that Dragonfly offers.

Additional context

Thank you for considering my request. I look forward to your response and any suggestions you might have.

[romange]

hi, what's the size of your workload? are you hosted on the cloud or use on-prem?

[totorofly]

hi, what's the size of your workload? are you hosted on the cloud or use on-prem?

Thank you for your prompt response.

I'm currently hosting my application on Alibaba Cloud, specifically using Kubernetes with a Redis cluster made up of 3 worker nodes. Each node has 8 CPU cores and 32 GB of memory. I have both RediSearch and RediJSON enabled on this setup.

As for the workload, my application handles approximately 2000 keys per second through the JSON.SET command, with each key being around 2000 bytes in size. Each key has a lifespan of around 8 minutes in Redis.

I hope this additional information provides a clearer picture of my current setup and requirements. Please let me know if there are any other details you need.

[romange]

Thanks!

1. Is 2k qps json.set is per single worker or overall?
2. what's the average ft.search query load?

Please note that Dragonfly does not support cluster configurations.

[totorofly]

Thanks!

1. Is 2k qps json.set is per single worker or overall?
2. what's the average ft.search query load?

Please note that Dragonfly does not support cluster configurations.

Thank you for your reply. Here are the answers to your questions:

1. The 2000 keys per second JSON.SET operation workload I mentioned is for the entire cluster, not per worker node.

2. For ft.search queries, during peak times, we have around 10-30 users per second, and each user's query involves approximately 10 ft.search operations. So the load for ft.search queries during peak times is about 100-300 queries per second.

As for the cluster configuration, my main focus is on improving write and query efficiency, so whether or not Dragonfly supports clustering is not a crucial factor for me. The key is performance. Currently, the high write load on my application is resulting in response times that exceed 1 second per user's query request, while our requirement is to respond to each user's query request within 500ms. If necessary, I am willing to scale up the cluster, but the single-threaded nature of Redis makes me feel that it's not utilizing the machine's performance to the fullest. Therefore, I am looking for a more optimal solution.

I hope this provides the necessary details for you. Please let me know if there's anything else you need.

[romange]

one last question. Can you put here an example ft.search query that you run on your database?

fyi, we currently do not support ft.search but we understand the community very much interested in it and we are considering to add it at least in some limited form. That's why I want to see if we can support your use-case.

500ms is a lot!

[totorofly]

one last question. Can you put here an example ft.search query that you run on your database?

fyi, we currently do not support ft.search but we understand the community very much interested in it and we are considering to add it at least in some limited form. That's why I want to see if we can support your use-case.

500ms is a lot!

1. Business Requirement: We provide consumers with the service of selecting and purchasing unique phone numbers.

2. User Scenario:

   • When a user clicks to search for a number on the frontend, a request is sent to us. The backend will then search in the dynamic number pool (nearly 1 million numbers and each number's TTL is 8 minutes) for 8-20 phone numbers that meet different search criteria and return them to the frontend for the user to choose from.
   • Each of the 8-20 slots has different matching rules. For example, the first slot will try to match numbers related to the user's birthday and the matching number must be the last 8 or 4 digits of the numbers in the number pool. For a user whose birthday is December 31, 1999, the priority of the ft.search query would be:
     • a. Look for numbers in the pool where the last 8 digits are 19991231.
     • b. If no numbers are found in step a, then ft.search for numbers where the last 4 digits are 1231.
     • c. If no numbers are found in step b, then ft.search for numbers where the last 4 digits are 1999.
     • d. If no numbers are found in step c, then ft.search for numbers where the middle 4 digits are 1231.
     • e. If no numbers are found in step d, then ft.search for numbers where the middle 4 digits are 1999.
     • f. If no numbers are found in step e, then ft.search for a number based on a fallback condition to ensure at least one number can be returned.
     • The above search conditions only need to meet one to return and will not continue to query. Please note that this is just the matching rule for one slot's return condition. According to the worst case, it needs to query 6 times. If 8 numbers are returned to the user, and each number needs to query 6 times, then a single user will use ft.search up to 48 times.

3. Implementation Logic:

   • To use the inverted index to quickly search for numbers that meet the conditions in the million-level number pool (the length of the number is 11 digits), we split each number in the number pool according to the last 8 digits, the last 6 digits, the middle 4 digits, the last 4 digits, the last 3 progress, whether it contains the birthday field, etc., assign meaning to the split numbers to form a json format split object, and establish an index for each key.

{
   "phone": "13033665786",
   "diffNumCount": 65,
   "status": "1",
   "owner": "",
   "ttlInSecond": 0,
   "preOrderTime": 0,
   "providerCode": "beijing",
   "touchCode": "0",
   "province": "beijing",
   "city": "beijing",
   "saleProvinceRange": "beijing",
   "saleCityRange": "beijing",
   "vendor": "2",
   "rule_common": {
       "prefixNum": "13X",
       "prefix3Bit": "130",
       "include4": "0",
       "hitMassRule": "1",
       "hitMassRuleId": "midAABB",
       "fiveNum": "0"
   },
   "rule_idCard": {
       "last4_yyyy": "5786",
       "last4_mmdd": "5786",
       "mid4_mmdd": "5786",
       "mid4_yyyy": "5786",
       "last4_idCardLast4": "5786",
       "mid4_idCardLast4": "5786",
       "last2_yy": "86"
   },
   "rule_car": {
       "any5": "65786",
       "any4": "X5786,6X786,65X86,657X6,6578X",
       "any3": "XX786,X5X86,X57X6,X578X,6XX86,6X7X6,6X78X,65XX6,65X8X,657XX",
       "tail5": "65786",
       "tail4": "5786",
       "tail3": "786",
       "continuous5": "66578,36657,33665,03366,30336,13033",
       "continuous4": "6578,6657,3665,3366,0336,3033,1303",
       "continuous3": "578,657,665,366,336,033,303,130"
   },
   "rule_mass": {
       "midAABB": "1",
       "anyAABB": "1",
       "anyXAA": "1",
       "anyABB": "1",
       "last_6": "1",
       "last689": "1",
       "head130": "1",
       "exclude_9": "1",
       "exclude_2": "1",
       "any66": "1"
   },
   "rule_continuity": {
       "continuity10": "1303366578,3033665786",
       "continuity9": "130336657,303366578,033665786",
       "continuity8": "13033665,30336657,03366578,33665786",
       "continuity7": "1303366,3033665,0336657,3366578,3665786",
       "continuity6": "130336,303366,033665,336657,366578,665786",
       "continuity5": "13033,30336,03366,33665,36657,66578,65786",
       "continuity4": "1303,3033,0336,3366,3665,6657,6578,5786",
       "continuity3": "130,303,033,336,366,665,657,578,786",
       "continuity2": "13,30,03,33,36,66,65,57,78,86"
   }
}

• When a single user requests, all numbers need to be fetched concurrently, so the implementation code snippet is as follows:

   func executeSlotSearch(ctx context.Context, index string, searchParams *RedisSearchParams, resultsMap map[int]SlotResult) error {
	defer func(t time.Time) {
		fmt.Println("executeSlotSearch:", time.Since(t))
	}(time.Now())
	var resultsMu sync.Mutex
	eg, ctx := errgroup.WithContext(ctx)

	for i := range searchParams.SlotSearchParams {
		slotIndex := i
		slotParams := searchParams.SlotSearchParams[i]

		eg.Go(func() error {
			remainingPositions := len(slotParams.PositionArr)
			options := buildOptions(index, slotParams, 0, int64(remainingPositions), searchParams.OccupiedTime)
			phoneRedisObjects, err := fetchPhoneAndUpdateWithOccupiedTime(ctx, options)
			if err != nil {
				fmt.Println("Error fetching and decoding phone Redis objects:", err)
			}

			resultsMu.Lock()
			currentSlotResult := resultsMap[slotIndex]
			currentSlotResult.PositionArr = slotParams.PositionArr
			if slotParams.IsFree {
				currentSlotResult.IsFree = slotParams.IsFree
			}

			currentSlotResult.PhoneRedisObjects = append(currentSlotResult.PhoneRedisObjects, phoneRedisObjects...)
			resultsMap[slotIndex] = currentSlotResult
			resultsMu.Unlock()
			return nil
		})
	}

	return eg.Wait()
}

• Example ft.search query:
  FT.SEARCH tm '@hitMassRuleId:lastABABAB|anyABCDABCD|lastAAABBB|lastAABBCC|lastABCABC|lastABCDDBCAXXX|anyAABBCC|anyAAABBB|lastAAAAB|lastAAAAA|anyABCDEF|anyAAAAA|lastAABBB|lastABCDABDCXXX|lastABCDBACD|lastABCDBACDXXX|lastABCDDCBA|lastAAAA|lastABCDACBDXXX|lastABBA|lastABBCBB|lastABCDABDC|anyAAAA|lastAABB|anyABABAB|anyABCABC|anyAAAAB|midAAAA|anyAAABB|anyABBCBB|lastABABtu368|anyAABBB|lastABBB|lastABABtu613|lastABAB|lastABABtu850|midBAAA|lastABCD|midAAAB|lastAABCC|midAABB|lastBrithYear758799|lastAAAB|midABCD|midABAB|any888|any666|lastAABAAXXX|lastABCCBAXX|anyABAB|anyABABtu368|anyBrithYear758799|anyABBA|anyAABB|anyABABtu850|anyABABtu613|lastABB|anyABCD|lastXAXAXAXA|lastXAXAXA|lastABAC|lastAXAXAX|anyAAA|head1889|lastABACAD|anyABBCDD|lastAXAXAXAX @ttlInSecond:[1683364311 +inf] @providerCode:beijing @status:1 @preOrderTime:[-inf (1683364009] @touchCode:P009 @province:beijing @city:beijing ' LIMIT 0 0

According to our actual operation, the response of a single ft.search is often just a few tens of milliseconds. However, because a single user needs to return 8-20 numbers, each number is called ft.search in parallel. As long as one of the numbers (slots) encounters the worst case and needs to query multiple times according to the designed query priority, then the time returned to the user has to be calculated based on the longest time of one of the slot number queries plus the network return time.

Although Redis Cluster provides horizontal scaling to improve write speed and query performance, personally, I do not favor this architecture philosophy that seems to waste machine resources. Therefore, I am looking for a better solution to implement the User Scenario.

[romange]

@dwzkit please dm me on discord

[romange]

Duplicate of #431. Please, see here for updates.