remove glog #44
Merged
remove glog #44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
artursouza
added a commit
that referenced
this pull request
Jun 25, 2024
* add back in scheduler app that I forgot to add in. fix go.mod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add a bunch of integration tests and run make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Convenient ignore for .local folder. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add scheduler to the list of binaries in Makefile. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move placement interface out of internal package. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move actor config out of internal package. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move actor api level out of internal. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add placement client to scheduler. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * update go.mod and rm the appID prefix from being returned to users Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix up some make lint issues Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make data to be bytes and add it to scheduler's trigger callback. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Triggers reminders using scheduler service. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Address comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Update pkg/actors/actors.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Revert Job's data back to Any. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * integration tests for daprd connecting to scheduler and performing crud ops on 10jobs. req validation Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add error integration tests for schedule job Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix imports and general cleanup and fix issue with require.Len on slice from PR review feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add helper for error reason string construction Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * build out scheduler server as grpc server for testing to force scheduler server errors. add tests for all other tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename constructReason file Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * metadata -> errMetadata to avoid confusion Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update tests/integration/suite/scheduler/api/jobs.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix names due to added PR feedback thru UI Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * incorporate Josh's feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update error strings based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Add etcd data dir, Listen addr, cleanup (#15) * pulling out changes from existing PRs to be cleaner Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cherrypick listen addr logic from other PR commit Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make scheduler host address singular Signed-off-by: joshvanl <me@joshvanl.dev> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * put back etcd name to fix err Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: joshvanl <me@joshvanl.dev> Co-authored-by: joshvanl <me@joshvanl.dev> * fix issue running tests in parallel by adding initial cluster info and more etcd config Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change scheduler logLevel back to info for test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * got ha for 2 schedulers working and am able to see the db in etcd via the cli Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move tests around based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move tests around based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup of comments and update test to schedule against a random scheduler, but ensure data exists on all schedulers Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup and switch initialCluster to []string Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update etcdClientPorts to be a map to de-dup the lookup logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework to have EtcdClientPort on scheduler for easier lookup later Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework etcd-client-ports Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix import Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add note to rm etcdctl cmd later Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm raw command for library func Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add initial connections pool for scheduler server of sidecars connecting Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * initial streaming from scheduler to sidecar at trigger time works. need to cleanup still and write tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup scheduler server.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix stream ctx to exit go routine properly Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * kill runtime go routine if ctx is cancelled or if it errs and pass err out of go routine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Create Scheduler Charts (#12) * initial charts. need to fix caching issue to confirm Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds ETCD data dir option, use emptyDir in statefulset Signed-off-by: joshvanl <me@joshvanl.dev> * fix healthz port Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates to charts Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip. need to fix connectivity issue, but added to injector and updated client code to be an abstract wrapper Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make scheduler host address singular Signed-off-by: joshvanl <me@joshvanl.dev> * Update cmd/scheduler/options/options.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> * rm reminders service name condition Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add listen address for scheduler Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rebase in trigger reminders via scheduler PR and fix actors scheduler client Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * ha -> replicaCount, update conditionals including it and use Etcd over ETCD vars Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm volumeClaimTemplates and cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix version issue Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm diff Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates to ensure ha for k8s works Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * condense var into 1 line Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: joshvanl <me@joshvanl.dev> * remove the sidecar connections from the connPool in scheduler when appropriate. general cleanup. add logs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * pkg/runtime/runtime.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * schedulerManager added, streaming works for n sidecars however, it doesn't stream back to the proper appID. this needs to be fixed Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * jobs now stream to the proper appID Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix runtime go routines to close properly. this schedules jobs properly to 2 sidecars of a diff appID Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * minConnCount -> max. fix lock on nsappid Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add backoff retry inf logic for sidecar connecting Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * random index for appID connection streaming choice and read lock for better efficiency Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * send only job data & metadata on stream. remove ConnectHosts. Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * string ptr for addresses to string. cleanup. add cmd options to testing framework for scheduler: placement, maxConns, maxWaitTime, add sidecar to notls test to get it to pass since scheduler needs a sidecar to stream back to at trigger time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make max sidecar conns -1 pending load tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * testing my push Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds closeCh to signal shutdown of watcher Signed-off-by: joshvanl <me@joshvanl.dev> * still need to polish code, but wanted to push bc its working for reconnecting and having sidecar wait for scheduler to come up Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update to latest cron lib. (#17) * Update to latest cron lib. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Implement DeleteReminder Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix IT for cron after internal schema change in etcd store. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix IT for scheduler. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Update cmd/daprd/options/options.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Use "@every" to convert period to schedule. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add GetJob again and assume namespace in metadata. Other comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Address comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * remove debug line. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Don't double close etcd. Signed-off-by: Artur Souza <asouza.pro@gmail.com> --------- Signed-off-by: Artur Souza <asouza.pro@gmail.com> Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> * streaming test ensuring triggered job logged on proper sidecar. adding wip rotate scheduler client if conn err. fixing to switch back to what I originally had. Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm unused rotate client on conn err. instead proceed to rotate schedulers per crud operation Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * gs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * unexport manager structs where possible. updates based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm \n Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Fix scheduler client ref. (#20) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * use runner manager for go routines for scheduler manager & refactor/update streaming code in scheduler manager to be cleaner Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update log spacing bc that was failing the streaming test. rm extra lock on pool Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update actor code to use schedulerManager for next client and rm schedulerClient Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update scheduler server conn pool to be in internal dir Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv struct creation to schedulerManager code, not runtime Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm maxConnsPerAppID and maxWaitTime Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add appID + ns as first class fields and comment out ttl since updating to the latest cron lib breaks it. this will be added by artur in a followup PR Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update notls test to rm time.sleep and assertEventually. update ns+appID to be in func and not caller logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm double string concatenation per josh PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm extra setting of scheduler addr on daprd in framework since its set higher up in exec area Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm unused var Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update trigger func to be latest lib Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * metadata of ns+appID added to scheduler logic, and removed from sidecar side Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm comment Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * simplify nextIdx for scheduler based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add code todos Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * pr feedback: watchJobs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv client funcs to client dir Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix weird spacing in proto file Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add schedulerManager Options struct Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * log.Errorf everywhere with addr Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * use atomic int instead of rand for next scheduler client. rm lock Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm SideCarConnDetails struct Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move ns, appID concat to pool pkg Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update pkg/scheduler/server/internal/pool.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * rm ctx per PR feedback since its unused Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * Update tests/integration/framework/process/daprd/options.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * added todo to rm appID + ns from test for streaming Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Productionises distributed scheduler (#22) * Fixes warning “Running http and grpc server on single port. This is not recommended for production.” Signed-off-by: Elena Kolevska <elena@kolevska.com> * Suffixes data dirs with instance id Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds space quota parameter Signed-off-by: Elena Kolevska <elena@kolevska.com> * Sets default quota to 2GB Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds compaction parameters Signed-off-by: Elena Kolevska <elena@kolevska.com> * Updates helm charts Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds namespace to data dir name. Renames etcdID to just ID. Signed-off-by: Elena Kolevska <elena@kolevska.com> --------- Signed-off-by: Elena Kolevska <elena@kolevska.com> * merge in master and comment out actorInternal in scheduler server.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Send Triggered Job to App (#23) * able to send triggered job back to app via the app channel from daprd sidecar using both grpc and http protocols Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change sidecar receiving job to debug level to still validate the scheduler stream Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * grpc test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * some cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework grpc app to add the OnJobEventFn and update test to use it. grpc appcallback test passes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip http test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * added http working test. need to make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update tests with stub for interface func for triggerJob to app now since its in the app channel interface Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * defer release of ch Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * go-etcd-cron (#25) * go-etcd-cron Signed-off-by: joshvanl <me@joshvanl.dev> * Fix multi-scheduler int test Signed-off-by: joshvanl <me@joshvanl.dev> * Review comments Signed-off-by: joshvanl <me@joshvanl.dev> * Rename schedule app job type to job Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * fix charts (#24) * restore test file diff, keep chart chagnes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix read-only err Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm space in go.mod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Bidirectional job trigger & ack. (#26) * Bidirectional job trigger & ack. Adds job ack from scheduler client for when job is finished processing and can be ticked. Adds mTLS authorization to scheduler API server. Adds integration tests for scheduler Jobs and Actor Reminders. Signed-off-by: joshvanl <me@joshvanl.dev> * Review comments & reconnect scheduler int test Signed-off-by: joshvanl <me@joshvanl.dev> * Update go-etcd-cron Signed-off-by: joshvanl <me@joshvanl.dev> * Linting Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Charts: Adds option to use dynamic PVC provistioned volume for Scheduler statefulset (#27) * Charts: Adds option to use PVC for Scheduler statefulset Adds optional `dapr_scheduler.cluster.persistentVolumeClaimName` helm chart values option to change the scheduler data dir volume to use the references PersistentVolumeClaim, rather than an empty dir, making ETCD data persistent across pod restarts. Also changes the volume and mount paths so that all schedulers share the same root mount path, but write to a sub directory of the form "/<namespace>/<scheduler-id>". Signed-off-by: joshvanl <me@joshvanl.dev> * Update scheduler volume to use volumeClaimTemplate Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * add opts.HealthzListenAddress to fix make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler e2e tests (#28) * merge & fix http status code check Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * triggered job e2e test for http app Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test iteration nums Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm time.sleep -> assert.eventually Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm local test changes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test name Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * grpc e2d works, need to cleanup grpc test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm grpc test and combine into http test. keep both apps tho. need to cleanup local test changes in scheduler_test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup local test changes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv things around Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * thread -> goroutine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update clients.go Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * rm commented line Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * PR review updates Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * review updates. add code todo Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> * Retry clients if first try fails (#29) * continuously retry scheduler clients if it fails upon the first try Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix indentation after UI committing Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * forgot to push this diff for e2e tests (#30) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler reminder preview feature (#31) * initial feature flag Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename protos. add getReminder scheduler logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix typos Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * undo exporting the internal struct. added and exported func to call for reminderPeriod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix typo Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * total credit for this code goes to @artursouza Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * test for actor reminder scheduler preview feature Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * v1.0 -> v1.0-alpha1 (#32) * v1.0 -> v1.0-alpha1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * v1.0-alpha1 for e2e app Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * revert some unintentional changes (#33) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * try to fix e2e (#34) * try to fix e2e Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm test changes since env var now Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds actor reminder scheduler perf tests (#35) * Adds actor reminder scheduler perf tests Signed-off-by: joshvanl <me@joshvanl.dev> * Run daprd in debug mode for test relying on new debug line Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Adds trigger performance tests (#36) Signed-off-by: joshvanl <me@joshvanl.dev> * update rpc names, fix mtls test (#37) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Fix scheduler int tests (#38) * fix idtypes test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks to fix some tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make tests pass consistently (#39) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix endpoint test (#40) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add ttl and up schedule time to see if it fixes the test (#41) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix %d -> %f for perf tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Updates go-etcd-cron to main to include dequeue delete (#42) Also updated the scheduler crud test to remove race condition on testing results but waiting longer to observe/not observe triggers. Signed-off-by: joshvanl <me@joshvanl.dev> * fix go.sum Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Feat dist scheduler perf tune (#43) * Increase dueTime for reminder Signed-off-by: joshvanl <me@joshvanl.dev> * Adds scheduler app as scope to config state store Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Tune V2 (#44) Signed-off-by: joshvanl <me@joshvanl.dev> * Enable HA for perf tests (#46) Signed-off-by: joshvanl <me@joshvanl.dev> * rm ha (#47) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Enable scheduler reminders in workflow performance test (#48) Signed-off-by: joshvanl <me@joshvanl.dev> * Enable scheduler reminders in workflow performance test (#49) Signed-off-by: joshvanl <me@joshvanl.dev> * Fix workflow perf config option (#50) Signed-off-by: joshvanl <me@joshvanl.dev> * Make runtime scheduler app channel use same interface. Set status code (#51) on HTTP send job to gRPC status code Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go-etcd-cron (#52) Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go.sum (#53) Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go-etcd-cron (#54) Signed-off-by: joshvanl <me@joshvanl.dev> * Fix buildx from arm64 host OS. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Force docker type for docker image. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Enable scheduler for workflows (#55) * Enable scheduler for workflows Enables scheduler to work with actor workflows. Updates go-etcd-cron library to fix job name validation errors, and ensure actor reminders are triggered on due_time, even when schedule set. Executes `actorRuntime` reminder execution handling in runtime scheduler client trigger handler, ensuring internal workflow actors are invoked. Advertises internal workflow `dapr.internal.default.xyz.workflow` and `dapr.internal.default.xyz.activity` actor types to scheduler so enable receiving workflow actor reminders. Adds scheduler workflow integration tests. Moves scheduler actor reminders as an implementation of the actor reminder interface, removing the no-op implementation. Signed-off-by: joshvanl <me@joshvanl.dev> * Adds namespace to scheduler advertised internal work actor type Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Adds CallActorReminder internal gRPC to hop reminder calls to (#57) correct host Signed-off-by: joshvanl <me@joshvanl.dev> * Always use localnamespace in acotr invocation, and use correct target (#58) AppID in reminder Signed-off-by: joshvanl <me@joshvanl.dev> * Restart scheduler between workflow runs (#59) Fix data race condition in remote scheduler integration test Signed-off-by: joshvanl <me@joshvanl.dev> * rename type->JobTargetMetadata per proposal feedback (#56) * rename type->kind per proposal feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * JobMetadataKind->JobTargetMetadata Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Test 100k Workflows with Scheduler (#60) * 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * just do the 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test.js Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler 100k workflows (#61) * 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * just do the 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test.js Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up timing and mem Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * 1k->100 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up all mem limits (#62) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * try rm-ing ctx timeout for actor reminder for workflow (#63) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler fix workflow perf (#65) * update existing scenarios to test 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * only do it for 2 of tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add rate (#66) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * 100k -> 10k (#67) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * upping/rm every timeout I could fine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up timeout (#68) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset workflow timeouts and vals/config (#69) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update charts/dapr/charts/dapr_scheduler/templates/dapr_scheduler_poddisruptionbudget.yaml Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> * Scheduler fix int tests: quorum/notls on windows (#64) * up time for windows int tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up int test times to pass Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update pkg/api/errors/errors.go Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * Update tests/perf/actor_reminder/actor_reminder_test.go Co-authored-by: Mike Nguyen <hey@mike.ee> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * gofmt Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Reset Workflow Config & Fix License (#71) * fix license yr & fireAt -> fire_at from PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix proto spacing and dont use empty Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update return type and rm deprecation warning Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset activity_actor.go diff Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix license and reset workflow config Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix flaky idtypes test, PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm legacy code bits per PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uint64 -> string per PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset listen-addr and uint64 type Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename scheduler api -> jobs api (#72) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update e2e app endpoint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler workflow perf test app + original workflow perf test app (#73) * add another app to test scheduler instead of replacing the existing one Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add : for new var Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * general cleanup of diff + grpc api -> jobs instead of scheduler Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * PR feedback updates + up test time (#74) * update rand -> crypto/rand Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update uuid -> id per PR feedback + up test time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uuid -> id, rand -> id++ Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uuid -> id Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up test time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change default listen addr, up timeouts Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset int test timeout Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm redundant err check, debug loglevel for notls- (I will revert this) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv err log to actual err conditional block Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add getJobFn Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * localhost -> 127.0.0.1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * trying @JoshVanL suggested changes to fix CI failures...TY Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweak notls: localhost->127.0.0.1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweak ctx handling. add debug loglevel (will revert) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * idtypes debug (will revert) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * notls debug loglevel again and 127.0.0.1 instead of localhost Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * acct for windows path being different in etcd. / vs \ Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * revert int test debug log level Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * log line added to idtypes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm log for testing Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm log levels Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Move OnJobEvent to OnJobEventAlpha1 (#75) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix pubsub E2E. (#76) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Artur Souza <asouza.pro@gmail.com> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Elena Kolevska <elena@kolevska.com> Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Artur Souza <asouza.pro@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com> Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Mike Nguyen <hey@mike.ee>
AnnuCode
pushed a commit
to AnnuCode/dapr
that referenced
this pull request
Aug 7, 2024
* add back in scheduler app that I forgot to add in. fix go.mod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add a bunch of integration tests and run make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Convenient ignore for .local folder. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add scheduler to the list of binaries in Makefile. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move placement interface out of internal package. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move actor config out of internal package. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move actor api level out of internal. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add placement client to scheduler. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * update go.mod and rm the appID prefix from being returned to users Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix up some make lint issues Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make data to be bytes and add it to scheduler's trigger callback. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Triggers reminders using scheduler service. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Address comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Update pkg/actors/actors.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Revert Job's data back to Any. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * integration tests for daprd connecting to scheduler and performing crud ops on 10jobs. req validation Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add error integration tests for schedule job Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix imports and general cleanup and fix issue with require.Len on slice from PR review feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add helper for error reason string construction Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * build out scheduler server as grpc server for testing to force scheduler server errors. add tests for all other tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename constructReason file Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * metadata -> errMetadata to avoid confusion Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update tests/integration/suite/scheduler/api/jobs.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix names due to added PR feedback thru UI Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * incorporate Josh's feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update error strings based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Add etcd data dir, Listen addr, cleanup (dapr#15) * pulling out changes from existing PRs to be cleaner Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cherrypick listen addr logic from other PR commit Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make scheduler host address singular Signed-off-by: joshvanl <me@joshvanl.dev> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * put back etcd name to fix err Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: joshvanl <me@joshvanl.dev> Co-authored-by: joshvanl <me@joshvanl.dev> * fix issue running tests in parallel by adding initial cluster info and more etcd config Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change scheduler logLevel back to info for test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * got ha for 2 schedulers working and am able to see the db in etcd via the cli Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move tests around based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move tests around based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup of comments and update test to schedule against a random scheduler, but ensure data exists on all schedulers Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup and switch initialCluster to []string Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update etcdClientPorts to be a map to de-dup the lookup logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework to have EtcdClientPort on scheduler for easier lookup later Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework etcd-client-ports Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix import Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add note to rm etcdctl cmd later Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm raw command for library func Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add initial connections pool for scheduler server of sidecars connecting Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * initial streaming from scheduler to sidecar at trigger time works. need to cleanup still and write tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup scheduler server.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix stream ctx to exit go routine properly Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * kill runtime go routine if ctx is cancelled or if it errs and pass err out of go routine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Create Scheduler Charts (dapr#12) * initial charts. need to fix caching issue to confirm Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds ETCD data dir option, use emptyDir in statefulset Signed-off-by: joshvanl <me@joshvanl.dev> * fix healthz port Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates to charts Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip. need to fix connectivity issue, but added to injector and updated client code to be an abstract wrapper Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make scheduler host address singular Signed-off-by: joshvanl <me@joshvanl.dev> * Update cmd/scheduler/options/options.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> * rm reminders service name condition Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add listen address for scheduler Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rebase in trigger reminders via scheduler PR and fix actors scheduler client Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * ha -> replicaCount, update conditionals including it and use Etcd over ETCD vars Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm volumeClaimTemplates and cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix version issue Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm diff Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates to ensure ha for k8s works Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * condense var into 1 line Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: joshvanl <me@joshvanl.dev> * remove the sidecar connections from the connPool in scheduler when appropriate. general cleanup. add logs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * pkg/runtime/runtime.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * schedulerManager added, streaming works for n sidecars however, it doesn't stream back to the proper appID. this needs to be fixed Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * jobs now stream to the proper appID Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix runtime go routines to close properly. this schedules jobs properly to 2 sidecars of a diff appID Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * minConnCount -> max. fix lock on nsappid Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add backoff retry inf logic for sidecar connecting Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * random index for appID connection streaming choice and read lock for better efficiency Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * send only job data & metadata on stream. remove ConnectHosts. Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * string ptr for addresses to string. cleanup. add cmd options to testing framework for scheduler: placement, maxConns, maxWaitTime, add sidecar to notls test to get it to pass since scheduler needs a sidecar to stream back to at trigger time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make max sidecar conns -1 pending load tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * testing my push Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds closeCh to signal shutdown of watcher Signed-off-by: joshvanl <me@joshvanl.dev> * still need to polish code, but wanted to push bc its working for reconnecting and having sidecar wait for scheduler to come up Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update to latest cron lib. (dapr#17) * Update to latest cron lib. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Implement DeleteReminder Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix IT for cron after internal schema change in etcd store. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix IT for scheduler. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Update cmd/daprd/options/options.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Use "@every" to convert period to schedule. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add GetJob again and assume namespace in metadata. Other comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Address comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * remove debug line. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Don't double close etcd. Signed-off-by: Artur Souza <asouza.pro@gmail.com> --------- Signed-off-by: Artur Souza <asouza.pro@gmail.com> Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> * streaming test ensuring triggered job logged on proper sidecar. adding wip rotate scheduler client if conn err. fixing to switch back to what I originally had. Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm unused rotate client on conn err. instead proceed to rotate schedulers per crud operation Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * gs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * unexport manager structs where possible. updates based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm \n Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Fix scheduler client ref. (dapr#20) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * use runner manager for go routines for scheduler manager & refactor/update streaming code in scheduler manager to be cleaner Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update log spacing bc that was failing the streaming test. rm extra lock on pool Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update actor code to use schedulerManager for next client and rm schedulerClient Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update scheduler server conn pool to be in internal dir Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv struct creation to schedulerManager code, not runtime Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm maxConnsPerAppID and maxWaitTime Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add appID + ns as first class fields and comment out ttl since updating to the latest cron lib breaks it. this will be added by artur in a followup PR Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update notls test to rm time.sleep and assertEventually. update ns+appID to be in func and not caller logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm double string concatenation per josh PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm extra setting of scheduler addr on daprd in framework since its set higher up in exec area Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm unused var Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update trigger func to be latest lib Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * metadata of ns+appID added to scheduler logic, and removed from sidecar side Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm comment Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * simplify nextIdx for scheduler based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add code todos Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * pr feedback: watchJobs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv client funcs to client dir Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix weird spacing in proto file Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add schedulerManager Options struct Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * log.Errorf everywhere with addr Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * use atomic int instead of rand for next scheduler client. rm lock Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm SideCarConnDetails struct Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move ns, appID concat to pool pkg Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update pkg/scheduler/server/internal/pool.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * rm ctx per PR feedback since its unused Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * Update tests/integration/framework/process/daprd/options.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * added todo to rm appID + ns from test for streaming Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Productionises distributed scheduler (dapr#22) * Fixes warning “Running http and grpc server on single port. This is not recommended for production.” Signed-off-by: Elena Kolevska <elena@kolevska.com> * Suffixes data dirs with instance id Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds space quota parameter Signed-off-by: Elena Kolevska <elena@kolevska.com> * Sets default quota to 2GB Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds compaction parameters Signed-off-by: Elena Kolevska <elena@kolevska.com> * Updates helm charts Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds namespace to data dir name. Renames etcdID to just ID. Signed-off-by: Elena Kolevska <elena@kolevska.com> --------- Signed-off-by: Elena Kolevska <elena@kolevska.com> * merge in master and comment out actorInternal in scheduler server.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Send Triggered Job to App (dapr#23) * able to send triggered job back to app via the app channel from daprd sidecar using both grpc and http protocols Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change sidecar receiving job to debug level to still validate the scheduler stream Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * grpc test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * some cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework grpc app to add the OnJobEventFn and update test to use it. grpc appcallback test passes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip http test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * added http working test. need to make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update tests with stub for interface func for triggerJob to app now since its in the app channel interface Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * defer release of ch Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * go-etcd-cron (dapr#25) * go-etcd-cron Signed-off-by: joshvanl <me@joshvanl.dev> * Fix multi-scheduler int test Signed-off-by: joshvanl <me@joshvanl.dev> * Review comments Signed-off-by: joshvanl <me@joshvanl.dev> * Rename schedule app job type to job Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * fix charts (dapr#24) * restore test file diff, keep chart chagnes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix read-only err Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm space in go.mod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Bidirectional job trigger & ack. (dapr#26) * Bidirectional job trigger & ack. Adds job ack from scheduler client for when job is finished processing and can be ticked. Adds mTLS authorization to scheduler API server. Adds integration tests for scheduler Jobs and Actor Reminders. Signed-off-by: joshvanl <me@joshvanl.dev> * Review comments & reconnect scheduler int test Signed-off-by: joshvanl <me@joshvanl.dev> * Update go-etcd-cron Signed-off-by: joshvanl <me@joshvanl.dev> * Linting Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Charts: Adds option to use dynamic PVC provistioned volume for Scheduler statefulset (dapr#27) * Charts: Adds option to use PVC for Scheduler statefulset Adds optional `dapr_scheduler.cluster.persistentVolumeClaimName` helm chart values option to change the scheduler data dir volume to use the references PersistentVolumeClaim, rather than an empty dir, making ETCD data persistent across pod restarts. Also changes the volume and mount paths so that all schedulers share the same root mount path, but write to a sub directory of the form "/<namespace>/<scheduler-id>". Signed-off-by: joshvanl <me@joshvanl.dev> * Update scheduler volume to use volumeClaimTemplate Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * add opts.HealthzListenAddress to fix make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler e2e tests (dapr#28) * merge & fix http status code check Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * triggered job e2e test for http app Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test iteration nums Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm time.sleep -> assert.eventually Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm local test changes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test name Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * grpc e2d works, need to cleanup grpc test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm grpc test and combine into http test. keep both apps tho. need to cleanup local test changes in scheduler_test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup local test changes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv things around Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * thread -> goroutine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update clients.go Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * rm commented line Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * PR review updates Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * review updates. add code todo Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> * Retry clients if first try fails (dapr#29) * continuously retry scheduler clients if it fails upon the first try Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix indentation after UI committing Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * forgot to push this diff for e2e tests (dapr#30) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler reminder preview feature (dapr#31) * initial feature flag Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename protos. add getReminder scheduler logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix typos Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * undo exporting the internal struct. added and exported func to call for reminderPeriod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix typo Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * total credit for this code goes to @artursouza Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * test for actor reminder scheduler preview feature Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * v1.0 -> v1.0-alpha1 (dapr#32) * v1.0 -> v1.0-alpha1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * v1.0-alpha1 for e2e app Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * revert some unintentional changes (dapr#33) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * try to fix e2e (dapr#34) * try to fix e2e Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm test changes since env var now Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds actor reminder scheduler perf tests (dapr#35) * Adds actor reminder scheduler perf tests Signed-off-by: joshvanl <me@joshvanl.dev> * Run daprd in debug mode for test relying on new debug line Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Adds trigger performance tests (dapr#36) Signed-off-by: joshvanl <me@joshvanl.dev> * update rpc names, fix mtls test (dapr#37) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Fix scheduler int tests (dapr#38) * fix idtypes test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks to fix some tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make tests pass consistently (dapr#39) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix endpoint test (dapr#40) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add ttl and up schedule time to see if it fixes the test (dapr#41) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix %d -> %f for perf tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Updates go-etcd-cron to main to include dequeue delete (dapr#42) Also updated the scheduler crud test to remove race condition on testing results but waiting longer to observe/not observe triggers. Signed-off-by: joshvanl <me@joshvanl.dev> * fix go.sum Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Feat dist scheduler perf tune (dapr#43) * Increase dueTime for reminder Signed-off-by: joshvanl <me@joshvanl.dev> * Adds scheduler app as scope to config state store Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Tune V2 (dapr#44) Signed-off-by: joshvanl <me@joshvanl.dev> * Enable HA for perf tests (dapr#46) Signed-off-by: joshvanl <me@joshvanl.dev> * rm ha (dapr#47) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Enable scheduler reminders in workflow performance test (dapr#48) Signed-off-by: joshvanl <me@joshvanl.dev> * Enable scheduler reminders in workflow performance test (dapr#49) Signed-off-by: joshvanl <me@joshvanl.dev> * Fix workflow perf config option (dapr#50) Signed-off-by: joshvanl <me@joshvanl.dev> * Make runtime scheduler app channel use same interface. Set status code (dapr#51) on HTTP send job to gRPC status code Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go-etcd-cron (dapr#52) Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go.sum (dapr#53) Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go-etcd-cron (dapr#54) Signed-off-by: joshvanl <me@joshvanl.dev> * Fix buildx from arm64 host OS. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Force docker type for docker image. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Enable scheduler for workflows (dapr#55) * Enable scheduler for workflows Enables scheduler to work with actor workflows. Updates go-etcd-cron library to fix job name validation errors, and ensure actor reminders are triggered on due_time, even when schedule set. Executes `actorRuntime` reminder execution handling in runtime scheduler client trigger handler, ensuring internal workflow actors are invoked. Advertises internal workflow `dapr.internal.default.xyz.workflow` and `dapr.internal.default.xyz.activity` actor types to scheduler so enable receiving workflow actor reminders. Adds scheduler workflow integration tests. Moves scheduler actor reminders as an implementation of the actor reminder interface, removing the no-op implementation. Signed-off-by: joshvanl <me@joshvanl.dev> * Adds namespace to scheduler advertised internal work actor type Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Adds CallActorReminder internal gRPC to hop reminder calls to (dapr#57) correct host Signed-off-by: joshvanl <me@joshvanl.dev> * Always use localnamespace in acotr invocation, and use correct target (dapr#58) AppID in reminder Signed-off-by: joshvanl <me@joshvanl.dev> * Restart scheduler between workflow runs (dapr#59) Fix data race condition in remote scheduler integration test Signed-off-by: joshvanl <me@joshvanl.dev> * rename type->JobTargetMetadata per proposal feedback (dapr#56) * rename type->kind per proposal feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * JobMetadataKind->JobTargetMetadata Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Test 100k Workflows with Scheduler (dapr#60) * 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * just do the 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test.js Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler 100k workflows (dapr#61) * 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * just do the 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test.js Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up timing and mem Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * 1k->100 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up all mem limits (dapr#62) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * try rm-ing ctx timeout for actor reminder for workflow (dapr#63) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler fix workflow perf (dapr#65) * update existing scenarios to test 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * only do it for 2 of tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add rate (dapr#66) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * 100k -> 10k (dapr#67) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * upping/rm every timeout I could fine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up timeout (dapr#68) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset workflow timeouts and vals/config (dapr#69) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update charts/dapr/charts/dapr_scheduler/templates/dapr_scheduler_poddisruptionbudget.yaml Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> * Scheduler fix int tests: quorum/notls on windows (dapr#64) * up time for windows int tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up int test times to pass Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update pkg/api/errors/errors.go Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * Update tests/perf/actor_reminder/actor_reminder_test.go Co-authored-by: Mike Nguyen <hey@mike.ee> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * gofmt Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Reset Workflow Config & Fix License (dapr#71) * fix license yr & fireAt -> fire_at from PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix proto spacing and dont use empty Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update return type and rm deprecation warning Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset activity_actor.go diff Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix license and reset workflow config Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix flaky idtypes test, PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm legacy code bits per PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uint64 -> string per PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset listen-addr and uint64 type Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename scheduler api -> jobs api (dapr#72) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update e2e app endpoint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler workflow perf test app + original workflow perf test app (dapr#73) * add another app to test scheduler instead of replacing the existing one Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add : for new var Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * general cleanup of diff + grpc api -> jobs instead of scheduler Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * PR feedback updates + up test time (dapr#74) * update rand -> crypto/rand Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update uuid -> id per PR feedback + up test time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uuid -> id, rand -> id++ Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uuid -> id Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up test time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change default listen addr, up timeouts Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset int test timeout Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm redundant err check, debug loglevel for notls- (I will revert this) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv err log to actual err conditional block Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add getJobFn Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * localhost -> 127.0.0.1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * trying @JoshVanL suggested changes to fix CI failures...TY Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweak notls: localhost->127.0.0.1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweak ctx handling. add debug loglevel (will revert) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * idtypes debug (will revert) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * notls debug loglevel again and 127.0.0.1 instead of localhost Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * acct for windows path being different in etcd. / vs \ Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * revert int test debug log level Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * log line added to idtypes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm log for testing Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm log levels Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Move OnJobEvent to OnJobEventAlpha1 (dapr#75) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix pubsub E2E. (dapr#76) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Artur Souza <asouza.pro@gmail.com> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Elena Kolevska <elena@kolevska.com> Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Artur Souza <asouza.pro@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com> Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Mike Nguyen <hey@mike.ee> Signed-off-by: Annu Singh <annu4444.as@gmail.com>
jake-engelberg
pushed a commit
to jake-engelberg/dapr
that referenced
this pull request
Sep 20, 2024
* add back in scheduler app that I forgot to add in. fix go.mod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add a bunch of integration tests and run make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Convenient ignore for .local folder. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add scheduler to the list of binaries in Makefile. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move placement interface out of internal package. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move actor config out of internal package. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Move actor api level out of internal. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add placement client to scheduler. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * update go.mod and rm the appID prefix from being returned to users Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix up some make lint issues Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make data to be bytes and add it to scheduler's trigger callback. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Triggers reminders using scheduler service. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Address comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Update pkg/actors/actors.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Revert Job's data back to Any. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * integration tests for daprd connecting to scheduler and performing crud ops on 10jobs. req validation Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add error integration tests for schedule job Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix imports and general cleanup and fix issue with require.Len on slice from PR review feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add helper for error reason string construction Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * build out scheduler server as grpc server for testing to force scheduler server errors. add tests for all other tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename constructReason file Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * metadata -> errMetadata to avoid confusion Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update tests/integration/suite/scheduler/api/jobs.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix names due to added PR feedback thru UI Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * incorporate Josh's feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update error strings based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Add etcd data dir, Listen addr, cleanup (dapr#15) * pulling out changes from existing PRs to be cleaner Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cherrypick listen addr logic from other PR commit Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make scheduler host address singular Signed-off-by: joshvanl <me@joshvanl.dev> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * put back etcd name to fix err Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: joshvanl <me@joshvanl.dev> Co-authored-by: joshvanl <me@joshvanl.dev> * fix issue running tests in parallel by adding initial cluster info and more etcd config Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change scheduler logLevel back to info for test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * got ha for 2 schedulers working and am able to see the db in etcd via the cli Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move tests around based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move tests around based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup of comments and update test to schedule against a random scheduler, but ensure data exists on all schedulers Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup and switch initialCluster to []string Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update etcdClientPorts to be a map to de-dup the lookup logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework to have EtcdClientPort on scheduler for easier lookup later Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework etcd-client-ports Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix import Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add note to rm etcdctl cmd later Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm raw command for library func Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add initial connections pool for scheduler server of sidecars connecting Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * initial streaming from scheduler to sidecar at trigger time works. need to cleanup still and write tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup scheduler server.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix stream ctx to exit go routine properly Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * kill runtime go routine if ctx is cancelled or if it errs and pass err out of go routine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Create Scheduler Charts (dapr#12) * initial charts. need to fix caching issue to confirm Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds ETCD data dir option, use emptyDir in statefulset Signed-off-by: joshvanl <me@joshvanl.dev> * fix healthz port Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates to charts Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip. need to fix connectivity issue, but added to injector and updated client code to be an abstract wrapper Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Make scheduler host address singular Signed-off-by: joshvanl <me@joshvanl.dev> * Update cmd/scheduler/options/options.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> * rm reminders service name condition Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add listen address for scheduler Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rebase in trigger reminders via scheduler PR and fix actors scheduler client Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * ha -> replicaCount, update conditionals including it and use Etcd over ETCD vars Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm volumeClaimTemplates and cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix version issue Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm diff Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * updates to ensure ha for k8s works Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * condense var into 1 line Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: joshvanl <me@joshvanl.dev> * remove the sidecar connections from the connPool in scheduler when appropriate. general cleanup. add logs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * pkg/runtime/runtime.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * schedulerManager added, streaming works for n sidecars however, it doesn't stream back to the proper appID. this needs to be fixed Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * jobs now stream to the proper appID Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix runtime go routines to close properly. this schedules jobs properly to 2 sidecars of a diff appID Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * minConnCount -> max. fix lock on nsappid Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add backoff retry inf logic for sidecar connecting Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * random index for appID connection streaming choice and read lock for better efficiency Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * send only job data & metadata on stream. remove ConnectHosts. Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * string ptr for addresses to string. cleanup. add cmd options to testing framework for scheduler: placement, maxConns, maxWaitTime, add sidecar to notls test to get it to pass since scheduler needs a sidecar to stream back to at trigger time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make max sidecar conns -1 pending load tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * testing my push Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds closeCh to signal shutdown of watcher Signed-off-by: joshvanl <me@joshvanl.dev> * still need to polish code, but wanted to push bc its working for reconnecting and having sidecar wait for scheduler to come up Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update to latest cron lib. (dapr#17) * Update to latest cron lib. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Implement DeleteReminder Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix IT for cron after internal schema change in etcd store. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix IT for scheduler. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Update cmd/daprd/options/options.go Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Use "@every" to convert period to schedule. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Add GetJob again and assume namespace in metadata. Other comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Address comments. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * remove debug line. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Don't double close etcd. Signed-off-by: Artur Souza <asouza.pro@gmail.com> --------- Signed-off-by: Artur Souza <asouza.pro@gmail.com> Co-authored-by: Cassie Coyle <cassie.i.coyle@gmail.com> * streaming test ensuring triggered job logged on proper sidecar. adding wip rotate scheduler client if conn err. fixing to switch back to what I originally had. Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm unused rotate client on conn err. instead proceed to rotate schedulers per crud operation Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * gs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * unexport manager structs where possible. updates based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm \n Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Fix scheduler client ref. (dapr#20) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * use runner manager for go routines for scheduler manager & refactor/update streaming code in scheduler manager to be cleaner Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update log spacing bc that was failing the streaming test. rm extra lock on pool Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update actor code to use schedulerManager for next client and rm schedulerClient Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update scheduler server conn pool to be in internal dir Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv struct creation to schedulerManager code, not runtime Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm maxConnsPerAppID and maxWaitTime Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add appID + ns as first class fields and comment out ttl since updating to the latest cron lib breaks it. this will be added by artur in a followup PR Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update notls test to rm time.sleep and assertEventually. update ns+appID to be in func and not caller logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm double string concatenation per josh PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm extra setting of scheduler addr on daprd in framework since its set higher up in exec area Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm unused var Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update trigger func to be latest lib Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * metadata of ns+appID added to scheduler logic, and removed from sidecar side Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm comment Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * simplify nextIdx for scheduler based on PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add code todos Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * pr feedback: watchJobs Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv client funcs to client dir Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix weird spacing in proto file Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add schedulerManager Options struct Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * log.Errorf everywhere with addr Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * use atomic int instead of rand for next scheduler client. rm lock Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm SideCarConnDetails struct Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * move ns, appID concat to pool pkg Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update pkg/scheduler/server/internal/pool.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * rm ctx per PR feedback since its unused Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * Update tests/integration/framework/process/daprd/options.go Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * added todo to rm appID + ns from test for streaming Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Productionises distributed scheduler (dapr#22) * Fixes warning “Running http and grpc server on single port. This is not recommended for production.” Signed-off-by: Elena Kolevska <elena@kolevska.com> * Suffixes data dirs with instance id Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds space quota parameter Signed-off-by: Elena Kolevska <elena@kolevska.com> * Sets default quota to 2GB Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds compaction parameters Signed-off-by: Elena Kolevska <elena@kolevska.com> * Updates helm charts Signed-off-by: Elena Kolevska <elena@kolevska.com> * Adds namespace to data dir name. Renames etcdID to just ID. Signed-off-by: Elena Kolevska <elena@kolevska.com> --------- Signed-off-by: Elena Kolevska <elena@kolevska.com> * merge in master and comment out actorInternal in scheduler server.go Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Send Triggered Job to App (dapr#23) * able to send triggered job back to app via the app channel from daprd sidecar using both grpc and http protocols Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change sidecar receiving job to debug level to still validate the scheduler stream Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * grpc test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * some cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test framework grpc app to add the OnJobEventFn and update test to use it. grpc appcallback test passes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * wip http test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * added http working test. need to make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update tests with stub for interface func for triggerJob to app now since its in the app channel interface Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * defer release of ch Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * go-etcd-cron (dapr#25) * go-etcd-cron Signed-off-by: joshvanl <me@joshvanl.dev> * Fix multi-scheduler int test Signed-off-by: joshvanl <me@joshvanl.dev> * Review comments Signed-off-by: joshvanl <me@joshvanl.dev> * Rename schedule app job type to job Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * fix charts (dapr#24) * restore test file diff, keep chart chagnes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix read-only err Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm space in go.mod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Bidirectional job trigger & ack. (dapr#26) * Bidirectional job trigger & ack. Adds job ack from scheduler client for when job is finished processing and can be ticked. Adds mTLS authorization to scheduler API server. Adds integration tests for scheduler Jobs and Actor Reminders. Signed-off-by: joshvanl <me@joshvanl.dev> * Review comments & reconnect scheduler int test Signed-off-by: joshvanl <me@joshvanl.dev> * Update go-etcd-cron Signed-off-by: joshvanl <me@joshvanl.dev> * Linting Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Charts: Adds option to use dynamic PVC provistioned volume for Scheduler statefulset (dapr#27) * Charts: Adds option to use PVC for Scheduler statefulset Adds optional `dapr_scheduler.cluster.persistentVolumeClaimName` helm chart values option to change the scheduler data dir volume to use the references PersistentVolumeClaim, rather than an empty dir, making ETCD data persistent across pod restarts. Also changes the volume and mount paths so that all schedulers share the same root mount path, but write to a sub directory of the form "/<namespace>/<scheduler-id>". Signed-off-by: joshvanl <me@joshvanl.dev> * Update scheduler volume to use volumeClaimTemplate Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * add opts.HealthzListenAddress to fix make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler e2e tests (dapr#28) * merge & fix http status code check Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * triggered job e2e test for http app Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test iteration nums Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm time.sleep -> assert.eventually Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm local test changes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test name Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * grpc e2d works, need to cleanup grpc test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm grpc test and combine into http test. keep both apps tho. need to cleanup local test changes in scheduler_test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup local test changes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv things around Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * thread -> goroutine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update clients.go Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * rm commented line Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * PR review updates Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * review updates. add code todo Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> * Retry clients if first try fails (dapr#29) * continuously retry scheduler clients if it fails upon the first try Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * fix indentation after UI committing Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * forgot to push this diff for e2e tests (dapr#30) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler reminder preview feature (dapr#31) * initial feature flag Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename protos. add getReminder scheduler logic Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix typos Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * undo exporting the internal struct. added and exported func to call for reminderPeriod Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix typo Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * total credit for this code goes to @artursouza Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * test for actor reminder scheduler preview feature Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * v1.0 -> v1.0-alpha1 (dapr#32) * v1.0 -> v1.0-alpha1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * v1.0-alpha1 for e2e app Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * revert some unintentional changes (dapr#33) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * try to fix e2e (dapr#34) * try to fix e2e Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm test changes since env var now Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Adds actor reminder scheduler perf tests (dapr#35) * Adds actor reminder scheduler perf tests Signed-off-by: joshvanl <me@joshvanl.dev> * Run daprd in debug mode for test relying on new debug line Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Adds trigger performance tests (dapr#36) Signed-off-by: joshvanl <me@joshvanl.dev> * update rpc names, fix mtls test (dapr#37) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Fix scheduler int tests (dapr#38) * fix idtypes test Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweaks to fix some tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make tests pass consistently (dapr#39) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix endpoint test (dapr#40) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add ttl and up schedule time to see if it fixes the test (dapr#41) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix %d -> %f for perf tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Updates go-etcd-cron to main to include dequeue delete (dapr#42) Also updated the scheduler crud test to remove race condition on testing results but waiting longer to observe/not observe triggers. Signed-off-by: joshvanl <me@joshvanl.dev> * fix go.sum Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Feat dist scheduler perf tune (dapr#43) * Increase dueTime for reminder Signed-off-by: joshvanl <me@joshvanl.dev> * Adds scheduler app as scope to config state store Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Tune V2 (dapr#44) Signed-off-by: joshvanl <me@joshvanl.dev> * Enable HA for perf tests (dapr#46) Signed-off-by: joshvanl <me@joshvanl.dev> * rm ha (dapr#47) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Enable scheduler reminders in workflow performance test (dapr#48) Signed-off-by: joshvanl <me@joshvanl.dev> * Enable scheduler reminders in workflow performance test (dapr#49) Signed-off-by: joshvanl <me@joshvanl.dev> * Fix workflow perf config option (dapr#50) Signed-off-by: joshvanl <me@joshvanl.dev> * Make runtime scheduler app channel use same interface. Set status code (dapr#51) on HTTP send job to gRPC status code Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go-etcd-cron (dapr#52) Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go.sum (dapr#53) Signed-off-by: joshvanl <me@joshvanl.dev> * Updates go-etcd-cron (dapr#54) Signed-off-by: joshvanl <me@joshvanl.dev> * Fix buildx from arm64 host OS. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Force docker type for docker image. Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Enable scheduler for workflows (dapr#55) * Enable scheduler for workflows Enables scheduler to work with actor workflows. Updates go-etcd-cron library to fix job name validation errors, and ensure actor reminders are triggered on due_time, even when schedule set. Executes `actorRuntime` reminder execution handling in runtime scheduler client trigger handler, ensuring internal workflow actors are invoked. Advertises internal workflow `dapr.internal.default.xyz.workflow` and `dapr.internal.default.xyz.activity` actor types to scheduler so enable receiving workflow actor reminders. Adds scheduler workflow integration tests. Moves scheduler actor reminders as an implementation of the actor reminder interface, removing the no-op implementation. Signed-off-by: joshvanl <me@joshvanl.dev> * Adds namespace to scheduler advertised internal work actor type Signed-off-by: joshvanl <me@joshvanl.dev> --------- Signed-off-by: joshvanl <me@joshvanl.dev> * Adds CallActorReminder internal gRPC to hop reminder calls to (dapr#57) correct host Signed-off-by: joshvanl <me@joshvanl.dev> * Always use localnamespace in acotr invocation, and use correct target (dapr#58) AppID in reminder Signed-off-by: joshvanl <me@joshvanl.dev> * Restart scheduler between workflow runs (dapr#59) Fix data race condition in remote scheduler integration test Signed-off-by: joshvanl <me@joshvanl.dev> * rename type->JobTargetMetadata per proposal feedback (dapr#56) * rename type->kind per proposal feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * JobMetadataKind->JobTargetMetadata Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Test 100k Workflows with Scheduler (dapr#60) * 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * just do the 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test.js Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler 100k workflows (dapr#61) * 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * just do the 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update test.js Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up timing and mem Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * 1k->100 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up all mem limits (dapr#62) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * try rm-ing ctx timeout for actor reminder for workflow (dapr#63) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler fix workflow perf (dapr#65) * update existing scenarios to test 100k workflows Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * only do it for 2 of tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add rate (dapr#66) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * 100k -> 10k (dapr#67) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * upping/rm every timeout I could fine Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up timeout (dapr#68) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset workflow timeouts and vals/config (dapr#69) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update charts/dapr/charts/dapr_scheduler/templates/dapr_scheduler_poddisruptionbudget.yaml Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> * Scheduler fix int tests: quorum/notls on windows (dapr#64) * up time for windows int tests Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up int test times to pass Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Update pkg/api/errors/errors.go Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * Update tests/perf/actor_reminder/actor_reminder_test.go Co-authored-by: Mike Nguyen <hey@mike.ee> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> * gofmt Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Reset Workflow Config & Fix License (dapr#71) * fix license yr & fireAt -> fire_at from PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix proto spacing and dont use empty Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update return type and rm deprecation warning Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset activity_actor.go diff Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix license and reset workflow config Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * fix flaky idtypes test, PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm legacy code bits per PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uint64 -> string per PR feedback Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset listen-addr and uint64 type Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rename scheduler api -> jobs api (dapr#72) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update e2e app endpoint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Scheduler workflow perf test app + original workflow perf test app (dapr#73) * add another app to test scheduler instead of replacing the existing one Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add : for new var Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * general cleanup of diff + grpc api -> jobs instead of scheduler Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * cleanup Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * PR feedback updates + up test time (dapr#74) * update rand -> crypto/rand Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * update uuid -> id per PR feedback + up test time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uuid -> id, rand -> id++ Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * uuid -> id Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * up test time Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * change default listen addr, up timeouts Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * reset int test timeout Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm redundant err check, debug loglevel for notls- (I will revert this) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * mv err log to actual err conditional block Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * add getJobFn Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * localhost -> 127.0.0.1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * trying @JoshVanL suggested changes to fix CI failures...TY Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweak notls: localhost->127.0.0.1 Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * tweak ctx handling. add debug loglevel (will revert) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * idtypes debug (will revert) Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * notls debug loglevel again and 127.0.0.1 instead of localhost Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * acct for windows path being different in etcd. / vs \ Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * revert int test debug log level Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * log line added to idtypes Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm log for testing Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * rm log levels Signed-off-by: Cassandra Coyle <cassie@diagrid.io> * Move OnJobEvent to OnJobEventAlpha1 (dapr#75) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * Fix pubsub E2E. (dapr#76) Signed-off-by: Artur Souza <asouza.pro@gmail.com> * make lint Signed-off-by: Cassandra Coyle <cassie@diagrid.io> --------- Signed-off-by: Cassandra Coyle <cassie@diagrid.io> Signed-off-by: Artur Souza <asouza.pro@gmail.com> Signed-off-by: Cassie Coyle <cassie.i.coyle@gmail.com> Signed-off-by: joshvanl <me@joshvanl.dev> Signed-off-by: Josh van Leeuwen <me@joshvanl.dev> Signed-off-by: Elena Kolevska <elena@kolevska.com> Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Artur Souza <asouza.pro@gmail.com> Co-authored-by: Josh van Leeuwen <me@joshvanl.dev> Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com> Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com> Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Mike Nguyen <hey@mike.ee> Signed-off-by: Jake Engelberg <jake@diagrid.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.