AWS SQS implementation

[elitemaks]

Continuing the work started by manoj-malviya - implementation of the queue based on Amazon SQS.
Currently, all commits by manoj-malviya are merged in this PR. I will continue to fix requests from old PR and glad to hear new feedbacks.

[zhuravljov]

@elitemaks, signal helper is deprecated. Please, merge master into the patch and see beanstalk driver as example for updating.

yii2-queue/src/drivers/beanstalk/Queue.php

Lines 42 to 70 in b2754ec

	/**
	* Listens queue and runs each job.
	*
	* @param bool $repeat whether to continue listening when queue is empty.
	* @param int $timeout number of seconds to wait for next message.
	* @return null|int exit code.
	* @internal for worker command only.
	* @since 2.0.2
	*/
	public function run($repeat, $timeout = 0)
	{
	return $this->runWorker(function (LoopInterface $loop) use ($repeat, $timeout) {
	while ($loop->canContinue()) {
	if ($payload = $this->getPheanstalk()->reserveFromTube($this->tube, $timeout)) {
	$info = $this->getPheanstalk()->statsJob($payload);
	if ($this->handleMessage(
	$payload->getId(),
	$payload->getData(),
	$info->ttr,
	$info->reserves
	)) {
	$this->getPheanstalk()->delete($payload);
	}
	} elseif (!$repeat) {
	break;
	}
	}
	});
	}

yii2-queue/src/drivers/beanstalk/Command.php

Lines 48 to 57 in b2754ec

	/**
	* Runs all jobs from beanstalk-queue.
	* It can be used as cron job.
	*
	* @return null|int exit code.
	*/
	public function actionRun()
	{
	return $this->queue->run(false);
	}

yii2-queue/src/drivers/beanstalk/Command.php

Lines 59 to 69 in b2754ec

	/**
	* Listens beanstalk-queue and runs new jobs.
	* It can be used as demon process.
	*
	* @param int $timeout number of seconds to wait a job.
	* @return null|int exit code.
	*/
	public function actionListen($timeout = 3)
	{
	return $this->queue->run(true, $timeout);
	}

Also, it should be covered with tests.

[elitemaks]

@zhuravljov Thanks, already implemented correct "run" and "listen" methods, please check.
Regarding tests - will work on them soon.

[zhuravljov]

Also consider ability to clear a queue and remove separate messages by ID in case this is possible.

yii2-queue/src/drivers/redis/Command.php

Lines 76 to 82 in f5ff542

	public function actionClear()
	{
	if ($this->confirm('Are you sure?')) {
	$this->queue->clear();
	$this->stdout("Queue has been cleared.\n");
	}
	}

http://docs.aws.amazon.com/aws-sdk-php/v2/api/class-Aws.Sqs.SqsClient.html#_purgeQueue

[zhuravljov]

Use long polling instead of sleep.
http://docs.aws.amazon.com/aws-sdk-php/v2/guide/service-sqs.html#receiving-messages

[elitemaks]

Done, please check. The limitation is that the max time long polling keeps the connection is 20 secs, so after that, the loop will be restarted. And, it works differently from "sleep" in a way, that if the message appears in a queue, it's handled without waiting for polling timeout.

[zhuravljov]

Use reserve inside getPayload.

[elitemaks]

Not sure about that. I think it's better to separate responsibility of these methods, as we can, for example, get payload and then delete the message, without the need of reserving it.

[zhuravljov]

In which cases this->getClient()->sendMessage() can return null instead of array?
If this is possible, pushMessage must throw an exception, because it always returns message ID. Else the block should be removed.

[elitemaks]

Correct, due to the documentation it returns model only. Exceptions are raised in aws sdk methods. I removed If block.

[elitemaks]

@zhuravljov "clear" has been implemented. Unfortunately, as I got it, there is no way to delete a specific message by ID. We can only get the next message in a queue and delete it - I think it does not make sense.

[thiagotalma]

Unfortunately, it is not possible to retrieve or remove a message from the queue based on an ID.
It is a limitation of SQS.

@zhuravljov This PR looks like it's good enough already.
Are you planning to release a new version when you accept it?

[elitemaks]

@thiagotalma still unit tests and documentation should be done.

@zhuravljov I started with unit tests, please check the last commit. But I need some assistance - with SQS almost everything depends on AWS client, so I had to mock it. Not sure if this is the best way, and if I need to write tests for other functions, as there is no way to check a real queue.

[elitemaks]

Created eng and rus docs for SQS queue driver.
@zhuravljov please review my edits for your review and the PR in general and let me know what should be done for its merging.

[elitemaks]

@samdark Hello Alex, please advise how to proceed with this PR as I did not get any reply form reviewers for a long time already.

[thiagotalma]

@zhuravljov That PR looks good enough already.
Do you have any predictions of when it will be accepted?

[zhuravljov]

@elitemaks would be nice to create tests with real connection to AWS, if possible. If not, it will without tests.

[elitemaks]

@zhuravljov Thanks for the reply! it's possible, but this way "key" and "secret" for account access will be stored in the extension's code. I can create a test acc and use those credentials. Please let me know if that's ok, then I can do it.

[elitemaks]

@zhuravljov The only way to include real AWS SQS account into tests is to put credentials to the config file. I think it's not correct to include creds in public repo.
Even if I find a way not to provide creds (like setup anonymous access or so) - SQS is a paid service so I'm not going to use my account for that. Yii team can create it but I don't think it worth it. So I think the easiest way, for now, is to go without tests including real SQS account.

[makasim]

It is possible to create a secret env var on travis side, no one can see it once stored and only build run by owners can use them. enqueue/sqs is tested that way

[elitemaks]

Ok but still, someone has to create that account on SQS.

[makasim]

That should be done by repo owners.

[mkubenka]

I would suggest to use ElasticMQ to mock SQS communication.

[zhuravljov]

Merged. Thank you!