Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate PRIMARY during concurrency #294

Closed
jakechan2012 opened this issue Mar 6, 2019 · 9 comments

Comments

@jakechan2012
Copy link

commented Mar 6, 2019

fail: DotNetCore.CAP.DefaultSubscriberExecutor[0]
      An exception occurred while executing the subscription method. Topic:events.activity_create, Id:1103205991968251904
DotNetCore.CAP.Internal.SubscriberExecutionFailedException: Duplicate entry '1103205992291213312' for key 'PRIMARY' ---> MySql.Data.MySqlClient.MySqlException: Duplicate entry '1103205992291213312' for key 'PRIMARY' ---> MySql.Data.MySqlClient.MySqlException: Duplicate entry '1103205992291213312' for key 'PRIMARY'
   at MySqlConnector.Core.ServerSession.TryAsyncContinuation(Task`1 task) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ServerSession.cs:line 1248
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\Core\ResultSet.cs:line 42
   --- End of inner exception stack trace ---
   at MySql.Data.MySqlClient.MySqlDataReader.ActivateResultSet(ResultSet resultSet) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 80
   at MySql.Data.MySqlClient.MySqlDataReader.ReadFirstResultSetAsync(IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 302
   at MySql.Data.MySqlClient.MySqlDataReader.CreateAsync(MySqlCommand command, CommandBehavior behavior, ResultSetProtocol resultSetProtocol, IOBehavior ioBehavior) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlDataReader.cs:line 287
   at MySqlConnector.Core.TextCommandExecutor.ExecuteReaderAsync(String commandText, MySqlParameterCollection parameterCollection, CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\Core\TextCommandExecutor.cs:line 37
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken) in C:\projects\mysqlconnector\src\MySqlConnector\MySql.Data.MySqlClient\MySqlCommand.cs:line 261
   at Dapper.SqlMapper.ExecuteImplAsync(IDbConnection cnn, CommandDefinition command, Object param) in C:\projects\dapper\Dapper\SqlMapper.Async.cs:line 678
   at DotNetCore.CAP.MySql.MySqlPublisher.ExecuteAsync(CapPublishedMessage message, ICapTransaction transaction, CancellationToken cancel)
   at DotNetCore.CAP.Abstractions.CapPublisherBase.PublishAsyncInternal(CapPublishedMessage message)
   at DotNetCore.CAP.MySql.MySqlPublisher.PublishCallbackAsync(CapPublishedMessage message)
   at DotNetCore.CAP.Internal.CallbackMessageSender.SendAsync(String messageId, String topicName, Object bodyObj)
   at DotNetCore.CAP.DefaultSubscriberExecutor.InvokeConsumerMethodAsync(CapReceivedMessage receivedMessage)
   --- End of inner exception stack trace ---
   at DotNetCore.CAP.DefaultSubscriberExecutor.InvokeConsumerMethodAsync(CapReceivedMessage receivedMessage)
   at DotNetCore.CAP.DefaultSubscriberExecutor.ExecuteWithoutRetryAsync(CapReceivedMessage message)
@jakechan2012

This comment has been minimized.

Copy link
Author

commented Mar 6, 2019

这个情况应该是我的应用负载均衡,但共用一个数据库引起的,但是 Subscriber 无法多线程订阅,事件排队执行肯定无法满足我的性能需求,我只能通过启动多个进程来解决,配置不同数据库还是加表前缀对我的部署都是一个挑战 ☹️

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Mar 6, 2019

Please provide your configuration and environment

@jakechan2012

This comment has been minimized.

Copy link
Author

commented Mar 6, 2019

@yang-xiaodong 我现在通过环境变量为每个应用分配了表前缀,并发下没有再出现主键问题

配置如下:

cap.UseMySql(mysql =>
{
    mysql.ConnectionString = options.MySqlConnectionString;
    mysql.TableNamePrefix = options.MySqlTableNamePrefix;
});
cap.UseRabbitMQ(mq =>
{
    mq.ExchangeName = options.RabbitMQExchangeName;
    mq.HostName = options.RabbitMQHostName;
    mq.UserName = options.RabbitMQUserName;
    mq.Password = options.RabbitMQPassword;
});
cap.DefaultGroup = options.Group;
cap.Version = options.Version;
cap.FailedRetryCount = options.FailedRetryCount;
cap.FailedRetryInterval = options.FailedRetryInterval;
cap.FailedThresholdCallback = options.FailedCallback;
cap.UseDashboard();

服务器是 Ubuntu Server 16.04, 8GB, i5

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

which version are u using?

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Mar 7, 2019

We use the snowflake algorithm to generate the primary key, you can find the code here, we randomize the workerId and datacenterId , the probability of theoretical conflict is very low, but this possibility is not ruled out

@jakechan2012

This comment has been minimized.

Copy link
Author

commented Mar 7, 2019

@yang-xiaodong the latest version: 2.4.2

yang-xiaodong added a commit that referenced this issue Mar 15, 2019
@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Mar 15, 2019

Hello @jakechan2012 , we will fixed the issus at v2.5.0

You can set up the snowflake algorithm generation by adding CAP_WORKERID and CAP_DATACENTERID to the environment variables to avoid concurrency conflicts.

if (!int.TryParse(Environment.GetEnvironmentVariable("CAP_WORKERID", EnvironmentVariableTarget.Machine), out var workerId))
{
workerId = random.Next((int)MaxWorkerId);
}
if (!int.TryParse(Environment.GetEnvironmentVariable("CAP_DATACENTERID", EnvironmentVariableTarget.Machine), out var datacenterId))
{
datacenterId = random.Next((int)MaxDatacenterId);
}

@jakechan2012

This comment has been minimized.

Copy link
Author

commented Mar 15, 2019

@yang-xiaodong many thx, but I prefer the option TableNamePrefix (both need set up env variables)

@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Mar 30, 2019

Fixed in v2.5.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.