Skip to content

Set concurrency limit#77

Merged
Wwwsylvia merged 18 commits intoAzure:mainfrom
Wwwsylvia:lixlei/fix-concurrency
Jan 27, 2021
Merged

Set concurrency limit#77
Wwwsylvia merged 18 commits intoAzure:mainfrom
Wwwsylvia:lixlei/fix-concurrency

Conversation

@Wwwsylvia
Copy link
Contributor

Refactor the worker package to limit amount of concurrent DELETE requests to ACR server

@Wwwsylvia Wwwsylvia changed the title Lixlei/fix concurrency Set concurrency limit Jan 12, 2021
// NewPurger creates a new Purger.
func NewPurger(batchSize int, acrClient api.AcrCLIClientInterface) *Purger {
procNum := int64(runtime.GOMAXPROCS(0))
worker := newPurgeWorker(acrClient)
Copy link
Member

@shizhMSFT shizhMSFT Jan 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPurgeWorker [](start = 11, length = 14)

It is not a worker. It is a job #Resolved

@shizhMSFT
Copy link
Member

shizhMSFT commented Jan 13, 2021

type PurgeJob struct {

Can you divide it into two structs as PurgeTagJob and PurgeManifestJob, having a common PurgeJobBase and matches a PurgeJob interface? #Resolved


Refers to: cmd/worker/purge_structures.go:9 in 3a71145. [](commit_id = 3a71145, deletion_comment = False)

cmd/acr/purge.go Outdated
if wErr.Error != nil {
return -1, wErr.Error
// jobs to be finished before continuing.
p.Wait(ctx)
Copy link
Member

@shizhMSFT shizhMSFT Jan 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be a dead lock if there are more tags than the batch size. #Resolved

workerPool: workerPool,
acrClient: acrClient,
wg: sync.WaitGroup{},
}
Copy link
Member

@shizhMSFT shizhMSFT Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: no need to initialize #Resolved

func (p *Purger) StartPurgeManifest(ctx context.Context, loginURL string, repoName string, digest string) {
job := newPurgeManifestJob(loginURL, repoName, digest)

p.workerPool.start(ctx, job, p.acrClient, &p.wg)
Copy link
Member

@shizhMSFT shizhMSFT Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

&p.wg [](start = 43, length = 5)

Please keep the wait group inside purger and not pass to the worker pool #Resolved

cmd/acr/purge.go Outdated
cmd.Flags().IntVar(&purgeParams.keep, "keep", 0, "Number of latest to-be-deleted tags to keep, use this when you want to keep at least x number of latest tags that could be deleted meeting all other filter criteria")
cmd.Flags().StringArrayVarP(&purgeParams.filters, "filter", "f", nil, "Specify the repository and a regular expression filter for the tag name, if a tag matches the filter and is older than the duration specified in ago it will be deleted")
cmd.Flags().StringArrayVarP(&purgeParams.configs, "config", "c", nil, "Authentication config paths (e.g. C://Users/docker/config.json)")
cmd.Flags().IntVarP(&purgeParams.taskNumber, "task-number", "t", 0, taskNumberDescription)
Copy link
Member

@shizhMSFT shizhMSFT Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 [](start = 66, length = 1)

default value should be set here. #Resolved

err = purger.FlushErrChan()
if err != nil {
return -1, err
}
Copy link
Member

@shizhMSFT shizhMSFT Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need to re-design this part. The objective is that we should return error if there is an error. #Resolved

Copy link
Member

@shizhMSFT shizhMSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🕐

@Wwwsylvia
Copy link
Contributor Author

@shizhMSFT Please help to review, Thanks!


// pool manages a limited number of workers that process purgeJob.
type pool struct {
size int
Copy link
Member

@shizhMSFT shizhMSFT Jan 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size int [](start = 1, length = 8)

It seems redundant. #Resolved

Copy link
Member

@shizhMSFT shizhMSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@Wwwsylvia Wwwsylvia merged commit a17c6b8 into Azure:main Jan 27, 2021
@Wwwsylvia Wwwsylvia deleted the lixlei/fix-concurrency branch January 27, 2021 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants