Use :latest task definition instead of registering a new one every time.

[pguartes-endava]

Jenkins and plugins versions report

Environment

Jenkins: 2.332.3
OS: Linux - 4.14.275-207.503.amzn2.x86_64
---
ace-editor:1.1
amazon-ecs:1.40
ant:1.13
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df
bitbucket:223.vd12f2bca5430
blueocean:1.25.3
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.25.3
blueocean-commons:1.25.3
blueocean-config:1.25.3
blueocean-core-js:1.25.3
blueocean-dashboard:1.25.3
blueocean-display-url:2.4.1
blueocean-events:1.25.3
blueocean-git-pipeline:1.25.3
blueocean-github-pipeline:1.25.3
blueocean-i18n:1.25.3
blueocean-jwt:1.25.3
blueocean-personalization:1.25.3
blueocean-pipeline-api-impl:1.25.3
blueocean-pipeline-editor:1.25.3
blueocean-pipeline-scm-api:1.25.3
blueocean-rest:1.25.3
blueocean-rest-impl:1.25.3
blueocean-web:1.25.3
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-6
bouncycastle-api:2.25
branch-api:2.7.0
build-timeout:1.20
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
cloudbees-bitbucket-branch-source:757.vddedc5f2589a_
cloudbees-folder:6.708.ve61636eb_65a_5
command-launcher:1.6
configuration-as-code:1414.v878271fc496f
credentials:1087.1089.v2f1b_9a_b_040e4
credentials-binding:1.27.1
display-url-api:2.3.5
durable-task:493.v195aefbb0ff2
echarts-api:5.3.0-2
email-ext:2.87
favorite:2.4.1
font-awesome-api:6.0.0-1
git:4.10.3
git-client:3.11.0
git-parameter:0.9.16
git-server:1.10
github:1.34.3
github-api:1.301-378.v9807bd746da5
github-branch-source:2.11.4
gradle:1.38
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-1.0
htmlpublisher:1.29
jackson2-api:2.13.1-246.va8a9f3eaf46a
javax-activation-api:1.2.0-2
javax-mail-api:1.6.2-5
jaxb:2.3.0.1
jdk-tool:1.5
jenkins-design-language:1.25.3
jjwt-api:0.11.2-9.c8b45b8bb173
jnr-posix-api:3.1.7-2
jquery:1.12.4-1
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.54
ldap:2.8
locale:144.v1a_998824ddb_3
lockable-resources:2.14
mailer:408.vd726a_1130320
matrix-auth:3.1
matrix-project:1.20
mercurial:2.16
momentjs:1.1.1
okhttp-api:4.9.3-105.vb96869f8ac3a
pam-auth:1.7
pipeline-aws:1.43
pipeline-build-step:2.16
pipeline-github-lib:36.v4c01db_ca_ed16
pipeline-graph-analysis:188.v3a01e7973f2c
pipeline-input-step:446.vf27b_0b_83500e
pipeline-milestone-step:100.v60a_03cd446e1
pipeline-model-api:1.9.3
pipeline-model-definition:1.9.3
pipeline-model-extensions:1.9.3
pipeline-rest-api:2.23
pipeline-stage-step:291.vf0a8a7aeeb50
pipeline-stage-tags-metadata:1.9.3
pipeline-stage-view:2.23
plain-credentials:1.8
plugin-util-api:2.14.0
popper-api:1.16.1-2
popper2-api:2.11.2-1
pubsub-light:1.16
resource-disposer:0.17
role-strategy:3.2.0
scm-api:595.vd5a_df5eb_0e39
script-security:1138.v8e727069a_025
snakeyaml-api:1.29.1
sonar:2.14
sse-gateway:1.25
ssh-credentials:1.19
ssh-slaves:1.806.v2253cedd3295
sshd:3.1.0
structs:308.v852b473a2b8c
timestamper:1.17
token-macro:280.v97a_82642793c
trilead-api:1.0.13
variant:1.4
workflow-aggregator:2.7
workflow-api:1138.v619fd5201b_2f
workflow-basic-steps:2.24
workflow-cps:2660.vb_c0412dc4e6d
workflow-cps-global-lib:566.vd0a_a_3334a_555
workflow-durable-task-step:1121.va_65b_d2701486
workflow-job:1145.v7f2433caa07f
workflow-multibranch:711.vdfef37cda_816
workflow-scm-step:2.13
workflow-step-api:622.vb_8e7c15b_c95a_
workflow-support:813.vb_d7c3d2984a_0
ws-cleanup:0.40

What Operating System are you using (both controller, and any agents involved in the problem)?

OS: Linux - 4.14.275-207.503.amzn2.x86_64

Reproduction steps

1. Configure ECS template with a template name

Expected Results

Plugin reuse the same task definition on ECS

Actual Results

The plugin creates one task definition per Job, which starts to increase the number of task definitions for the aws account. Currently we have thousands of revisions of the same task definition.

Anything else?

We tried Task Definition Override option but it seems to work for that purpose but Jobs don't run in parallel. Is there another way?

[cmlara]

For those who come across this in the future, the following log lines are most relevant to determine why a task definition is being created

INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on container definition: true
INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on volumes: true
INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on task role: true
INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on execution role: true
INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on network mode: false

In my deployment the Match on network was always false when I utilized "default" for the "Network mode" (this is for a Fargate deployment)

Looking at

amazon-ecs-plugin/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSService.java

Line 281 in a3cffcb

templateMatchesExistingNetworkMode = null == currentTaskDefinition.getNetworkMode();

I do not believe that the test could ever return a TRUE result when using "default" as I do not believe getNetworkMode() can ever return NULL for an existing task definition.

Setting the "Network mode" to the appropriate network type instead of default solved the issue in my deployment. This may or may not be the cause for the OP.

[Sebasti1492]

Same issue. The task definition is created every time for each job though there is no change in the definition.
Log shows container definition no match however the container definition is still the same for the agent versions.
As per Jenkins logs as below:

2022-07-15 18:46:31.201+0000 [id=39]	INFO	c.c.j.plugins.amazonecs.ECSCloud#provision: Asked to provision 1 agent(s) for: aloc-linux-agent
2022-07-15 18:46:31.202+0000 [id=39]	INFO	c.c.j.plugins.amazonecs.ECSCloud#provision: Will provision app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q, for label: aloc-linux-agent
2022-07-15 18:46:41.250+0000 [id=33]	INFO	hudson.slaves.NodeProvisioner#update: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q provisioning successfully completed. We have now 19 computer(s)
2022-07-15 18:46:41.276+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSService#findTaskDefinition: No existing task definition found for family or ARN: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent
2022-07-15 18:46:41.394+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:1,ContainerDefinitions: [{Name: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,Image: <accntid>.dkr.ecr.us-east-1.amazonaws.com/app-uai2005906-trs-jenkins-nonprod-linux-agent:RELEASE,Cpu: 2048,MemoryReservation: 4096,Links: [],PortMappings: [],Essential: true,EntryPoint: [/jenkins_exec/jenkins-agent],Command: [],Environment: [{Name: JENKINS_AGENT_WORKDIR,Value: /jenkins_home/jenkins}, {Name: JENKINS_URL,Value: https://jenkins-dev.xxxxx.xxx.com}],EnvironmentFiles: [],MountPoints: [{SourceVolume: jenkins-efs,ContainerPath: /jenkins_home,ReadOnly: false}],VolumesFrom: [],Secrets: [],DependsOn: [],Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],LogConfiguration: {LogDriver: awslogs,Options: {awslogs-group=/aws/ecs/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-ecs-task, awslogs-region=us-east-1, awslogs-stream-prefix=ecs-task},SecretOptions: []},SystemControls: [],ResourceRequirements: [],}],Family: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,TaskRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-linux-agent-ecs-task-role,ExecutionRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-ecs-task-exec-role,NetworkMode: awsvpc,Revision: 1,Volumes: [{Name: jenkins-efs,EfsVolumeConfiguration: {FileSystemId: fs-0d90702807251532d,RootDirectory: /,TransitEncryption: ENABLED,AuthorizationConfig: {AccessPointId: fsap-0c57d79981273aeb6,Iam: ENABLED}},}],Status: ACTIVE,RequiresAttributes: [{Name: com.amazonaws.ecs.capability.logging-driver.awslogs,}, {Name: ecs.capability.execution-role-awslogs,}, {Name: ecs.capability.efsAuth,}, {Name: com.amazonaws.ecs.capability.ecr-auth,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.19,}, {Name: ecs.capability.efs,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.21,}, {Name: com.amazonaws.ecs.capability.task-iam-role,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.25,}, {Name: ecs.capability.execution-role-ecr-pull,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.18,}, {Name: ecs.capability.task-eni,}],PlacementConstraints: [],Compatibilities: [EC2, FARGATE],RuntimePlatform: {CpuArchitecture: X86_64,OperatingSystemFamily: LINUX},RequiresCompatibilities: [FARGATE],Cpu: 2048,Memory: 4096,InferenceAccelerators: [],RegisteredAt: Fri Jul 15 18:46:41 UTC 2022,RegisteredBy: arn:aws:sts::<accntid>:assumed-role/app-uai2005906-trs-jenkins-nonprod-master-ecs-task-role/c42850fa9db249d7828d31e6faf649de,}
2022-07-15 18:46:41.394+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Starting agent with task definition arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:1}
2022-07-15 18:46:42.076+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Agent started with task arn : arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/6672adb7700040d783f7642fd4075cdc
2022-07-15 18:46:42.076+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: TaskArn: arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/6672adb7700040d783f7642fd4075cdc
2022-07-15 18:46:42.076+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:1
2022-07-15 18:46:42.076+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: ClusterArn: arn:aws:ecs:us-east-1:<accntid>:cluster/app-uai2005906-trs-jenkins-nonprod-ecs-cluster
2022-07-15 18:46:42.076+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: ContainerInstanceArn: null
2022-07-15 18:46:42.153+0000 [id=619]	INFO	hudson.model.AsyncAperiodicWork#lambda$doAperiodicRun$1: Started Update IdP Metadata from URL PeriodicWork
2022-07-15 18:46:42.153+0000 [id=619]	INFO	hudson.model.AsyncAperiodicWork#lambda$doAperiodicRun$1: Finished Update IdP Metadata from URL PeriodicWork. 0 ms
2022-07-15 18:48:13.310+0000 [id=41]	INFO	o.j.p.P.u.DropCachePeriodicWork#doRun: begin schedule clean...
2022-07-15 18:48:13.310+0000 [id=41]	INFO	o.j.p.P.u.DropCachePeriodicWork#doRun: end schedule clean...
2022-07-15 18:49:08.817+0000 [id=682]	INFO	h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /x.x.x.x:34922
2022-07-15 18:49:21.201+0000 [id=40]	INFO	c.c.j.plugins.amazonecs.ECSCloud#provision: Asked to provision 1 agent(s) for: aloc-linux-agent
2022-07-15 18:49:21.202+0000 [id=40]	INFO	c.c.j.plugins.amazonecs.ECSCloud#provision: Will provision app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h, for label: aloc-linux-agent
2022-07-15 18:49:31.244+0000 [id=42]	INFO	hudson.slaves.NodeProvisioner#update: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h provisioning successfully completed. We have now 20 computer(s)
2022-07-15 18:49:31.270+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on container definition: false
2022-07-15 18:49:31.270+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on volumes: true
2022-07-15 18:49:31.270+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on task role: true
2022-07-15 18:49:31.271+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on execution role: true
2022-07-15 18:49:31.271+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Match on network mode: true
2022-07-15 18:49:31.397+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSService#registerTemplate: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:2,ContainerDefinitions: [{Name: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,Image: <accntid>.dkr.ecr.us-east-1.amazonaws.com/app-uai2005906-trs-jenkins-nonprod-linux-agent:RELEASE,Cpu: 2048,MemoryReservation: 4096,Links: [],PortMappings: [],Essential: true,EntryPoint: [/jenkins_exec/jenkins-agent],Command: [],Environment: [{Name: JENKINS_AGENT_WORKDIR,Value: /jenkins_home/jenkins}, {Name: JENKINS_URL,Value: https://jenkins-dev.xxxxx.xxx.com}],EnvironmentFiles: [],MountPoints: [{SourceVolume: jenkins-efs,ContainerPath: /jenkins_home,ReadOnly: false}],VolumesFrom: [],Secrets: [],DependsOn: [],Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],LogConfiguration: {LogDriver: awslogs,Options: {awslogs-group=/aws/ecs/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-ecs-task, awslogs-region=us-east-1, awslogs-stream-prefix=ecs-task},SecretOptions: []},SystemControls: [],ResourceRequirements: [],}],Family: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,TaskRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-linux-agent-ecs-task-role,ExecutionRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-ecs-task-exec-role,NetworkMode: awsvpc,Revision: 2,Volumes: [{Name: jenkins-efs,EfsVolumeConfiguration: {FileSystemId: fs-0d90702807251532d,RootDirectory: /,TransitEncryption: ENABLED,AuthorizationConfig: {AccessPointId: fsap-0c57d79981273aeb6,Iam: ENABLED}},}],Status: ACTIVE,RequiresAttributes: [{Name: com.amazonaws.ecs.capability.logging-driver.awslogs,}, {Name: ecs.capability.execution-role-awslogs,}, {Name: ecs.capability.efsAuth,}, {Name: com.amazonaws.ecs.capability.ecr-auth,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.19,}, {Name: ecs.capability.efs,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.21,}, {Name: com.amazonaws.ecs.capability.task-iam-role,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.25,}, {Name: ecs.capability.execution-role-ecr-pull,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.18,}, {Name: ecs.capability.task-eni,}],PlacementConstraints: [],Compatibilities: [EC2, FARGATE],RuntimePlatform: {CpuArchitecture: X86_64,OperatingSystemFamily: LINUX},RequiresCompatibilities: [FARGATE],Cpu: 2048,Memory: 4096,InferenceAccelerators: [],RegisteredAt: Fri Jul 15 18:49:31 UTC 2022,RegisteredBy: arn:aws:sts::<accntid>:assumed-role/app-uai2005906-trs-jenkins-nonprod-master-ecs-task-role/c42850fa9db249d7828d31e6faf649de,}
2022-07-15 18:49:31.397+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: Starting agent with task definition arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:2}
2022-07-15 18:49:32.206+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: Agent started with task arn : arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/e820fea89540483199cf135c67fc787e
2022-07-15 18:49:32.207+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: TaskArn: arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/e820fea89540483199cf135c67fc787e
2022-07-15 18:49:32.207+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:2
2022-07-15 18:49:32.207+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: ClusterArn: arn:aws:ecs:us-east-1:<accntid>:cluster/app-uai2005906-trs-jenkins-nonprod-ecs-cluster
2022-07-15 18:49:32.207+0000 [id=694]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: ContainerInstanceArn: null
2022-07-15 18:49:42.200+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Task started, waiting for agent to become online
2022-07-15 18:49:42.201+0000 [id=546]	INFO	c.c.j.p.amazonecs.ECSLauncher#waitForAgent: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Agent connected

[helloPiers]

I notice two other cases where the matching fails, falsely:

1. If the container definition contains environment variables, these are not received in any defined order from ECS API, so most often the container definition is deemed different, even though it's the same env pairs, just in a different order
2. The automatic tags "jenkins.label" and "jenkins.templatename" are not added to the "wanted" list of tags before comparing with the previously registered template's tags (which does contain those), so it always thinks the tags are different. (It's likely the same sort order thing applies here, too.)

[helloPiers]

Actually, checking the code, the second of the above cases isn't actually effective, because although the tags are checked and difference (incorrectly) reported, no further reference to that difference is made in the code afterwards.

(templateTagsMatchesExistingTags is set, logged, but not tested in the if later.)

[pluralistix]

Hey,

we have created aws/aws-sdk-java#3043 for it and also opened a support case with AWS.

First saw https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html "Revisions per task definition family: 1,000,000" and thought....Pff...And then

❯ aws ecs list-task-definitions --family-prefix jenkins-build-cluster-xxx --status ACTIVE --max-items 2147483647 | jq -r '.taskDefinitionArns|length'
725010

Oh... 😄 (or 😱 - depends)...

BR