feat: use sort set to impl queue

[killagu]

Use sort set to keep queue in order and keep same value only insert once

[fengmk2]

有什么文章看看是什么原理么

[killagu]

https://redis.io/commands/zadd/
https://redis.io/commands/zpopmin/
https://redis.io/commands/zcount/

这三个可以看一下。

[fengmk2]

除了保证同步任务不重复，后面还是需要考虑真的有并发同步执行的时候，数据更新的原子性，目前可以先忽略。

[fengmk2]

oss 并发写入按道理是不会有问题的。

[codecov]

Codecov Report

Merging #277 (b40d954) into main (269cbf1) will decrease coverage by 0.40%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #277      +/-   ##
==========================================
- Coverage   97.43%   97.02%   -0.41%     
==========================================
  Files         119      119              
  Lines        3661     3665       +4     
  Branches      330      330              
==========================================
- Hits         3567     3556      -11     
- Misses         94      109      +15     

Impacted Files	Coverage Δ
app/core/service/TaskService.ts	97.05% <ø> (-1.48%)	⬇️
app/infra/QueueAdapter.ts	100.00% <100.00%> (ø)
config/config.default.ts	75.00% <0.00%> (-15.63%)	⬇️
app/infra/NFSClientAdapter.ts	74.07% <0.00%> (-14.82%)	⬇️
app/common/adapter/binary/PlaywrightBinary.ts	94.11% <0.00%> (-5.89%)	⬇️
app/port/controller/BinarySyncController.ts	92.30% <0.00%> (-5.13%)	⬇️
app/common/adapter/NFSAdapter.ts	96.66% <0.00%> (-3.34%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[Zheaoli]

First, Redis is not a good choice for the task queue. I think it would be better to use MQ or other tools.

Second, maybe you can use use Lua script to keep the command atomic

[killagu]

MQ is good for queue, but we choice less infrastructure dependency so use the redis.

[killagu]

Yes, lua script is better, we should use it.

[killagu]

And infra can be replace with other implement. It's ok to use mq impl the new queue adapter.

[killagu]

According to the redis doc.

Prior versions of Redis made scripting available only via the EVAL command, which allows a Lua script to be sent for execution by the server. The core use cases for Eval Scripts is executing part of your application logic inside Redis, efficiently and atomically. Such script can perform conditional updates across multiple keys, possibly combining several different data types.

Using EVAL requires that the application sends the entire script for execution every time. Because this results in network and script compilation overheads, Redis provides an optimization in the form of the EVALSHA command. By first calling SCRIPT LOAD to obtain the script's SHA1, the application can invoke it repeatedly afterward with its digest alone.

By design, Redis only caches the loaded scripts. That means that the script cache can become lost at any time, such as after calling SCRIPT FLUSH, after restarting the server, or when failing over to a replica. The application is responsible for reloading scripts during runtime if any are missing. The underlying assumption is that scripts are a part of the application and not maintained by the Redis server.

This approach suits many light-weight scripting use cases, but introduces several difficulties once an application becomes complex and relies more heavily on scripting, namely:

All client application instances must maintain a copy of all scripts. That means having some mechanism that applies script updates to all of the application's instances.
Calling cached scripts within the context of a transaction increases the probability of the transaction failing because of a missing script. Being more likely to fail makes using cached scripts as building blocks of workflows less attractive.
SHA1 digests are meaningless, making debugging the system extremely hard (e.g., in a MONITOR session).
When used naively, EVAL promotes an anti-pattern in which scripts the client application renders verbatim scripts instead of responsibly using the KEYS and ARGV Lua APIs.
Because they are ephemeral, a script can't call another script. This makes sharing and reusing code between scripts nearly impossible, short of client-side preprocessing (see the first point).
To address these needs while avoiding breaking changes to already-established and well-liked ephemeral scripts, Redis v7.0 introduces Redis Functions.

Use lua script with eval may encounter some scripts lost problem, and use function will be better. But we use the aliyun redis is v5, so we can not use it for now.

Strict order is not necessary for the task queue, it's ok to impl queue with two step.

[fengmk2]

1.10.0

[Zheaoli]

According to the redis doc.

Prior versions of Redis made scripting available only via the EVAL command, which allows a Lua script to be sent for execution by the server. The core use cases for Eval Scripts is executing part of your application logic inside Redis, efficiently and atomically. Such script can perform conditional updates across multiple keys, possibly combining several different data types.

Using EVAL requires that the application sends the entire script for execution every time. Because this results in network and script compilation overheads, Redis provides an optimization in the form of the EVALSHA command. By first calling SCRIPT LOAD to obtain the script's SHA1, the application can invoke it repeatedly afterward with its digest alone.

By design, Redis only caches the loaded scripts. That means that the script cache can become lost at any time, such as after calling SCRIPT FLUSH, after restarting the server, or when failing over to a replica. The application is responsible for reloading scripts during runtime if any are missing. The underlying assumption is that scripts are a part of the application and not maintained by the Redis server.

This approach suits many light-weight scripting use cases, but introduces several difficulties once an application becomes complex and relies more heavily on scripting, namely:

All client application instances must maintain a copy of all scripts. That means having some mechanism that applies script updates to all of the application's instances.

Calling cached scripts within the context of a transaction increases the probability of the transaction failing because of a missing script. Being more likely to fail makes using cached scripts as building blocks of workflows less attractive.

SHA1 digests are meaningless, making debugging the system extremely hard (e.g., in a MONITOR session).

When used naively, EVAL promotes an anti-pattern in which scripts the client application renders verbatim scripts instead of responsibly using the KEYS and ARGV Lua APIs.

Because they are ephemeral, a script can't call another script. This makes sharing and reusing code between scripts nearly impossible, short of client-side preprocessing (see the first point).

To address these needs while avoiding breaking changes to already-established and well-liked ephemeral scripts, Redis v7.0 introduces Redis Functions.

Use lua script with eval may encounter some scripts lost problem, and use function will be better. But we use the aliyun redis is v5, so we can not use it for now.

Strict order is not necessary for the task queue, it's ok to impl queue with two step.

Actually, the script-losing problem because the client use the cached script. BTW, this PR is LGTM for me if we can assume that we don't need atomic action here.