Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable Standalone User Event Dev mode #4680

Merged
merged 2 commits into from Oct 15, 2019
Merged

Enable Standalone User Event Dev mode #4680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
248d5c9
Add user event dev mode
chetanmeh Oct 10, 2019
1c69aa1
Apply suggestions from code review
chetanmeh Oct 15, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
37 changes: 21 additions & 16 deletions core/standalone/README.md
View file
Open in desktop
Expand Up @@ -93,23 +93,28 @@ $ java -jar openwhisk-standalone.jar -h
-d, --data-dir <arg> Directory used for storage
--dev-kcf Enables KubernetesContainerFactory for local
development
--dev-mode Developer mode speeds up the startup by
disabling preflight checks and avoiding
explicit pulls.
--disable-color-logging Disables colored logging
--kafka-docker-port <arg> Kafka port for use by docker based services.
If not specified then 9091 or some random
free port (if 9091 is busy) would be used
--kafka-port <arg> Kafka port. If not specified then 9092 or
some random free port (if 9092 is busy) would
be used
-p, --port <arg> Server port
--dev-mode Developer mode speeds up the startup by
disabling preflight checks and avoiding
explicit pulls.
--dev-user-events-port <arg> Specify the port for the user-event
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to have optional and default to 9095? If user-events is run with an IDE, it will start on this port by default.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to support that but scallop arg parser does not support it. If I specify --dev-user-events-port then need to provide an explicit value otherwise it gives an error like below

[openwhisk] Error: Bad arguments for option 'dev-user-events-port': '' - you should provide exactly one argument for this option

service. This mode can be used for local
development of user-event service by
configuring Prometheus to connect to
existing running service instance
--disable-color-logging Disables colored logging
--kafka-docker-port <arg> Kafka port for use by docker based
services. If not specified then 9091 or
some random free port (if 9091 is busy)
would be used
--kafka-port <arg> Kafka port. If not specified then 9092 or
some random free port (if 9092 is busy)
would be used
-p, --port <arg> Server port
-v, --verbose
--zk-port <arg> Zookeeper port. If not specified then 2181 or
some random free port (if 2181 is busy) would
be used
-h, --help Show help message
--version Show version of this program
--zk-port <arg> Zookeeper port. If not specified then 2181
or some random free port (if 2181 is busy)
would be used
-h, --help Show help message

OpenWhisk standalone server
```
Expand Down
10 changes: 8 additions & 2 deletions core/standalone/src/main/scala/org/apache/openwhisk/standalone/StandaloneOpenWhisk.scala
View file
Open in desktop
Expand Up @@ -101,6 +101,10 @@ class Conf(arguments: Seq[String]) extends ScallopConf(Conf.expandAllMode(argume

val devKcf = opt[Boolean](descr = "Enables KubernetesContainerFactory for local development")

val devUserEventsPort = opt[Int](
descr = "Specify the port for the user-event service. This mode can be used for local " +
"development of user-event service by configuring Prometheus to connect to existing running service instance")

mainOptions = Seq(manifest, configFile, apiGw, couchdb, userEvents, kafka, kafkaUi)

verify()
Expand Down Expand Up @@ -219,7 +223,8 @@ object StandaloneOpenWhisk extends SLF4JLogging {

val couchSvcs = if (conf.couchdb()) Some(startCouchDb(dataDir, dockerClient)) else None
val userEventSvcs =
if (conf.userEvents()) startUserEvents(conf.port(), kafkaDockerPort, workDir, dataDir, dockerClient)
if (conf.userEvents() || conf.devUserEventsPort.isSupplied)
startUserEvents(conf.port(), kafkaDockerPort, conf.devUserEventsPort.toOption, workDir, dataDir, dockerClient)
else Seq.empty

val svcs = Seq(defaultSvcs, apiGwSvcs, couchSvcs.toList, kafkaSvcs, userEventSvcs).flatten
Expand Down Expand Up @@ -503,6 +508,7 @@ object StandaloneOpenWhisk extends SLF4JLogging {

private def startUserEvents(owPort: Int,
kafkaDockerPort: Int,
existingUserEventSvcPort: Option[Int],
workDir: File,
dataDir: File,
dockerClient: StandaloneDockerClient)(
Expand All @@ -511,7 +517,7 @@ object StandaloneOpenWhisk extends SLF4JLogging {
ec: ExecutionContext,
materializer: ActorMaterializer): Seq[ServiceContainer] = {
implicit val tid: TransactionId = TransactionId(systemPrefix + "userevents")
val k = new UserEventLauncher(dockerClient, owPort, kafkaDockerPort, workDir, dataDir)
val k = new UserEventLauncher(dockerClient, owPort, kafkaDockerPort, existingUserEventSvcPort, workDir, dataDir)

val f = k.run()
val g = f.andThen {
Expand Down
44 changes: 27 additions & 17 deletions core/standalone/src/main/scala/org/apache/openwhisk/standalone/UserEventLauncher.scala
View file
Open in desktop
Expand Up @@ -29,15 +29,19 @@ import pureconfig.loadConfigOrThrow

import scala.concurrent.{ExecutionContext, Future}

class UserEventLauncher(docker: StandaloneDockerClient, owPort: Int, kafkaDockerPort: Int, workDir: File, dataDir: File)(
implicit logging: Logging,
ec: ExecutionContext,
actorSystem: ActorSystem,
materializer: ActorMaterializer,
tid: TransactionId) {
class UserEventLauncher(docker: StandaloneDockerClient,
owPort: Int,
kafkaDockerPort: Int,
existingUserEventSvcPort: Option[Int],
workDir: File,
dataDir: File)(implicit logging: Logging,
ec: ExecutionContext,
actorSystem: ActorSystem,
materializer: ActorMaterializer,
tid: TransactionId) {

//owPort+1 is used by Api Gateway
private val userEventPort = checkOrAllocatePort(owPort + 2)
private val userEventPort = existingUserEventSvcPort.getOrElse(checkOrAllocatePort(owPort + 2))
private val prometheusPort = checkOrAllocatePort(9090)
private val grafanaPort = checkOrAllocatePort(3000)

Expand All @@ -60,15 +64,21 @@ class UserEventLauncher(docker: StandaloneDockerClient, owPort: Int, kafkaDocker
}

def runUserEvents(): Future[ServiceContainer] = {
val env = Map("KAFKA_HOSTS" -> s"$hostIp:$kafkaDockerPort")

logging.info(this, s"Starting User Events: $userEventPort")
val name = containerName("user-events")
val params = Map("-p" -> Set(s"$userEventPort:9095"))
val args = createRunCmd(name, env, params)

val f = docker.runDetached(userEventConfig.image, args, true)
f.map(_ => ServiceContainer(userEventPort, s"http://localhost:$userEventPort", name))
existingUserEventSvcPort match {
case Some(_) =>
logging.info(this, s"Connecting to pre existing user-event service at $userEventPort")
Future.successful(ServiceContainer(userEventPort, s"http://localhost:$userEventPort", "Existing user-event"))
case None =>
val env = Map("KAFKA_HOSTS" -> s"$hostIp:$kafkaDockerPort")

logging.info(this, s"Starting User Events: $userEventPort")
val name = containerName("user-events")
val params = Map("-p" -> Set(s"$userEventPort:9095"))
val args = createRunCmd(name, env, params)

val f = docker.runDetached(userEventConfig.image, args, true)
f.map(_ => ServiceContainer(userEventPort, s"http://localhost:$userEventPort", name))
}
}

def runPrometheus(): Future[(StandaloneDockerContainer, ServiceContainer)] = {
Expand Down Expand Up @@ -107,7 +117,7 @@ class UserEventLauncher(docker: StandaloneDockerClient, owPort: Int, kafkaDocker

val volParams = Map(
"-v" -> Set(
s"${grafanaDataDir.getAbsolutePath}:/var/lib/grafanas",
s"${grafanaDataDir.getAbsolutePath}:/var/lib/grafana",
s"${grafanaConfigDir.getAbsolutePath}/provisioning/:/etc/grafana/provisioning/",
s"${grafanaConfigDir.getAbsolutePath}/dashboards/:/var/lib/grafana/dashboards/"))
val name = containerName("grafana")
Expand Down