Skip to content
This repository has been archived by the owner on Oct 29, 2021. It is now read-only.

PROPOSAL: mds-core transition to OMF City Services Working group #244

Closed
jfh01 opened this issue Feb 5, 2020 · 2 comments · Fixed by #249
Closed

PROPOSAL: mds-core transition to OMF City Services Working group #244

jfh01 opened this issue Feb 5, 2020 · 2 comments · Fixed by #249

Comments

@jfh01
Copy link
Contributor

jfh01 commented Feb 5, 2020 •
edited

To date, mds-core has been developed primarily by LADOT and their contractor Ellis and Associations. As part of the code contribution from LADOT to OMF, it is intend that mds-core be transitioned to management by the OMF City Services Working Group. It is our hope that a community of developers and interested stakeholders can be built around mds-core, similar to the community that exists for the Mobility Data Specification itself. That community will drive future development of mds-core with continued contribution from LADOT and their contractors.

Here you will find a proposal to the community for how to accomplish this transition:

https://docs.google.com/presentation/d/1y8WzdK4o2iA_v57zoEMgIuyBWqhyp4VufENAUwFlK1s/edit#slide=id.g643cf86fc3_0_0

We are soliciting feedback on this proposal. It will be presented to the City Services Working Group on an upcoming call as well. Please leave questions, comments, and reactions in this issue, or feel free to email jascha@openmobilityfoundation.org directly.
@schnuerle
Copy link
Member

Work in progress. Follow along with PR #249 for merger of deployable and testable code to OMF organization.

@schnuerle schnuerle linked a pull request May 21, 2020 that will close this issue
Catch-up from lacuna-tech/mds-core #249
Merged
schnuerle pushed a commit that referenced this issue May 21, 2020
@avatarneil @janedotx
@macsj200 @jwtodd @BigIdeas @levi217 @alexgottscha @kemiller @evanxadkins @hunterowens @twelch
Catch-up from lacuna-tech/mds-core (#249)
f2e8a81 
*Autogenerated Release notes, commits, authors from Squash:*

* [mds-metrics] Fix binning bug (#70)

* Fix weird array construction bug in metrics API binning method

* Remove async modifier

* Cleanup and add state_snapshot test

* Cleanup

* Bump root version

* Publish

 - @container-images/env-inject@0.1.21
 - @container-images/mds-agency@0.1.22
 - @container-images/mds-audit@0.1.23
 - @container-images/mds-compliance@0.1.22
 - @container-images/mds-daily@0.1.22
 - @container-images/mds-metrics-sheet@0.1.23
 - @container-images/mds-metrics@0.0.4
 - @container-images/mds-native@0.0.20
 - @container-images/mds-policy-author@0.1.22
 - @container-images/mds-policy@0.1.22
 - @container-images/mds-provider@0.1.22
 - @mds-core/mds-agency@0.0.24
 - @mds-core/mds-api-authorizer@0.1.22
 - @mds-core/mds-api-helpers@0.1.22
 - @mds-core/mds-api-server@0.1.22
 - @mds-core/mds-audit@0.1.34
 - @mds-core/mds-cache@0.1.22
 - @mds-core/mds-compliance@0.1.23
 - @mds-core/mds-daily@0.0.23
 - @mds-core/mds-db@0.1.22
 - @mds-core/mds-logger@0.1.20
 - @mds-core/mds-metrics-sheet@0.0.23
 - @mds-core/mds-metrics@0.0.4
 - @mds-core/mds-native@0.0.20
 - @mds-core/mds-policy-author@0.0.19
 - @mds-core/mds-policy@0.0.23
 - @mds-core/mds-provider@1.0.22
 - @mds-core/mds-providers@0.1.22
 - @mds-core/mds-stream@0.1.22
 - @mds-core/mds-test-data@0.1.22
 - @mds-core/mds-types@0.1.19
 - @mds-core/mds-utils@0.1.22

* Add some more logs for the Metrics API prototype (#73)

* Bump root package version

* Publish

 - @container-images/env-inject@0.1.22
 - @container-images/mds-agency@0.1.23
 - @container-images/mds-audit@0.1.24
 - @container-images/mds-compliance@0.1.23
 - @container-images/mds-daily@0.1.23
 - @container-images/mds-metrics-sheet@0.1.24
 - @container-images/mds-metrics@0.0.5
 - @container-images/mds-native@0.0.21
 - @container-images/mds-policy-author@0.1.23
 - @container-images/mds-policy@0.1.23
 - @container-images/mds-provider@0.1.23
 - @mds-core/mds-agency@0.0.25
 - @mds-core/mds-api-authorizer@0.1.23
 - @mds-core/mds-api-helpers@0.1.23
 - @mds-core/mds-api-server@0.1.23
 - @mds-core/mds-audit@0.1.35
 - @mds-core/mds-cache@0.1.23
 - @mds-core/mds-compliance@0.1.24
 - @mds-core/mds-daily@0.0.24
 - @mds-core/mds-db@0.1.23
 - @mds-core/mds-logger@0.1.21
 - @mds-core/mds-metrics-sheet@0.0.24
 - @mds-core/mds-metrics@0.0.5
 - @mds-core/mds-native@0.0.21
 - @mds-core/mds-policy-author@0.0.20
 - @mds-core/mds-policy@0.0.24
 - @mds-core/mds-provider@1.0.23
 - @mds-core/mds-providers@0.1.23
 - @mds-core/mds-stream@0.1.23
 - @mds-core/mds-test-data@0.1.23
 - @mds-core/mds-types@0.1.20
 - @mds-core/mds-utils@0.1.23

* [mds-db] Fix `editGeography` (#74)

* Fix `editGeography`.

* ts assist

* Fixed `editGeography` with TS assist.

* Bump root version

* Publish

 - @container-images/env-inject@0.1.23
 - @container-images/mds-agency@0.1.24
 - @container-images/mds-audit@0.1.25
 - @container-images/mds-compliance@0.1.24
 - @container-images/mds-daily@0.1.24
 - @container-images/mds-metrics-sheet@0.1.25
 - @container-images/mds-metrics@0.0.6
 - @container-images/mds-native@0.0.22
 - @container-images/mds-policy-author@0.1.24
 - @container-images/mds-policy@0.1.24
 - @container-images/mds-provider@0.1.24
 - @mds-core/mds-agency@0.0.26
 - @mds-core/mds-api-authorizer@0.1.24
 - @mds-core/mds-api-helpers@0.1.24
 - @mds-core/mds-api-server@0.1.24
 - @mds-core/mds-audit@0.1.36
 - @mds-core/mds-cache@0.1.24
 - @mds-core/mds-compliance@0.1.25
 - @mds-core/mds-daily@0.0.25
 - @mds-core/mds-db@0.1.24
 - @mds-core/mds-logger@0.1.22
 - @mds-core/mds-metrics-sheet@0.0.25
 - @mds-core/mds-metrics@0.0.6
 - @mds-core/mds-native@0.0.22
 - @mds-core/mds-policy-author@0.0.21
 - @mds-core/mds-policy@0.0.25
 - @mds-core/mds-provider@1.0.24
 - @mds-core/mds-providers@0.1.24
 - @mds-core/mds-stream@0.1.24
 - @mds-core/mds-test-data@0.1.24
 - @mds-core/mds-types@0.1.21
 - @mds-core/mds-utils@0.1.24

* fix error message (#77)

* mdscl/istio: install (un)install via helm straight up (#75)

* mdscl/istio: install (un)install via helm straight up

* mdsctl/istio: uninstallIstio cleanup

* [mds-audit] Support attaching media to audits (#76)

* Add new tables for audit attachments

* Integrate with audit attachments db tables

* New endpoints to add and delete attachments; return attachments with audits

* Lint

* Minor cleanups to s3 upload key and base url

* Move S3 config to env variables

* Let caller catch writeAttachment exception; type fixes

* Lint

* [mds-audit] Include initial provider state with audits returned via API (#42)

* Add MDS vehicle status and location to get audit response

* Add test for provider info in audit response

* Capitalize DESC keyword

* Add provider_event_time to Get Audit response

* Remove references to dropped columns from schema

* Use enums

* Fix import

* Make ORDER BY configurable in readEventsWithTelemetry

* Fix errors when running `yarn image`

* Use optional chaining from TS 3.7

* Bump root version

* Publish

 - @container-images/env-inject@0.1.24
 - @container-images/mds-agency@0.1.25
 - @container-images/mds-audit@0.1.26
 - @container-images/mds-compliance@0.1.25
 - @container-images/mds-daily@0.1.25
 - @container-images/mds-metrics-sheet@0.1.26
 - @container-images/mds-metrics@0.0.7
 - @container-images/mds-native@0.0.23
 - @container-images/mds-policy-author@0.1.25
 - @container-images/mds-policy@0.1.25
 - @container-images/mds-provider@0.1.25
 - @mds-core/mds-agency@0.0.27
 - @mds-core/mds-api-authorizer@0.1.25
 - @mds-core/mds-api-helpers@0.1.25
 - @mds-core/mds-api-server@0.1.25
 - @mds-core/mds-audit@0.1.37
 - @mds-core/mds-cache@0.1.25
 - @mds-core/mds-compliance@0.1.26
 - @mds-core/mds-daily@0.0.26
 - @mds-core/mds-db@0.1.25
 - @mds-core/mds-logger@0.1.23
 - @mds-core/mds-metrics-sheet@0.0.26
 - @mds-core/mds-metrics@0.0.7
 - @mds-core/mds-native@0.0.23
 - @mds-core/mds-policy-author@0.0.22
 - @mds-core/mds-policy@0.0.26
 - @mds-core/mds-provider@1.0.25
 - @mds-core/mds-providers@0.1.25
 - @mds-core/mds-stream@0.1.25
 - @mds-core/mds-test-data@0.1.25
 - @mds-core/mds-types@0.1.22
 - @mds-core/mds-utils@0.1.25

* Backend method for finding policies that use particular geographies. (#49)

* [mds-logger] Overloading log with log level parameter to support ternary conditions & fluent syntax (#71)

* Overloading log with log level parameter to support ternary conditions & fluent syntax

* remove psql-client (#84)

* need to have the domain inserted for both TLS and non-TLS (#83)

* allow additional hosts for egress if needed (#82)

* Add more granular readAllEvents performance logging (#80)

* [mds-schema-validators] Pull all JSON Schema-related definitions and code into a new package. (#78)

* Consolidating validator/schema-related code in mds-policy to mds-utils

* Consolidating validator/schema-related code in mds-compliance to mds-utils

* Consolidating validator/schema-related code in mds-compliance to mds-utils

* extracting JSON Schema definitions and code into mds-json-schema

* fixing build errors caused by moving validation to new package

* renaming mds-json-schema -> mds-schema-validators and cleaning up build issues

* moving ValidationError back to mds-utils

* removing hapi/joi from mds-utils

* updating tsconfigs

* Referencing RULE_TYPES when constructing schemas

* Revert "Referencing RULE_TYPES when constructing schemas"

This reverts commit 87def6e65910b44b3e632f6b3525bf993804882b.

* Referencing RULE_TYPES when constructing schemas

* Configuration Service/API (#91)

* Config service client and tests

* Config API and tests

* Config API container image

* Add debug configurations for config api/service

* create a configmap from files in helm/mds/mds-config, mount it to the mds-config pod under that path /mds-config

* Allow settings to use .json or .json5 extensions

* [mds-compliance] Compliance engine sorts vehicles by earliest telemetry data, so most … (#89)

* Compliance engine sorts vehicles by earliest telemetry data, so most recent events will be the ones tagged as violators.

* Updating sort function.

* Removing unnecessary spread operator.

* Fix helm configmap glob (#95)

* [mds-cache, mds-db] Instrument some cache and db operations with timing logs (#97)

* Instrument some cache and db operations with timing logs

* Add provider to getVehicles timing log

* Update dev/tooling dependencies. (#93)

* Upgrade to latest package dependency versions

* Clean up test scripts using config files

* Bump some coverage levels

* [mds-audit] Relax lookup VIN response requirements (#96)

* Relax lookup VIN response requirements

* Add logs for missing info; return telemetry when possible

* Bump root version

* Publish

 - @container-images/env-inject@0.1.25
 - @container-images/mds-agency@0.1.26
 - @container-images/mds-audit@0.1.27
 - @container-images/mds-compliance@0.1.26
 - @container-images/mds-config@0.0.2
 - @container-images/mds-daily@0.1.26
 - @container-images/mds-metrics-sheet@0.1.27
 - @container-images/mds-metrics@0.0.8
 - @container-images/mds-native@0.0.24
 - @container-images/mds-policy-author@0.1.26
 - @container-images/mds-policy@0.1.26
 - @container-images/mds-provider@0.1.26
 - @mds-core/mds-agency@0.0.28
 - @mds-core/mds-api-authorizer@0.1.26
 - @mds-core/mds-api-helpers@0.1.26
 - @mds-core/mds-api-server@0.1.26
 - @mds-core/mds-audit@0.1.38
 - @mds-core/mds-cache@0.1.26
 - @mds-core/mds-compliance@0.1.27
 - @mds-core/mds-config-service@0.0.2
 - @mds-core/mds-config@0.0.2
 - @mds-core/mds-daily@0.0.27
 - @mds-core/mds-db@0.1.26
 - @mds-core/mds-logger@0.1.24
 - @mds-core/mds-metrics-sheet@0.0.27
 - @mds-core/mds-metrics@0.0.8
 - @mds-core/mds-native@0.0.24
 - @mds-core/mds-policy-author@0.0.23
 - @mds-core/mds-policy@0.0.27
 - @mds-core/mds-provider@1.0.26
 - @mds-core/mds-providers@0.1.26
 - @mds-core/mds-schema-validators@0.1.2
 - @mds-core/mds-stream@0.1.26
 - @mds-core/mds-test-data@0.1.26
 - @mds-core/mds-types@0.1.23
 - @mds-core/mds-utils@0.1.26

* Fix debugging for unit tests (#105)

* mdsctl: prompt-to-proceed if not using docker-desktop (#85)

* mdsctl: prompt-to-proceed if not using docker-desktop

* mdsctl: bit of cleanup

* Add moped to VEHICLE_TYPES enum (#106)

* Support merging multiple settings properties (#108)

* Support merging multiple settings files

* Get settings property names from query string

* Support existing single settings property API

* [mds-agency] [mds-db] Initial Stop Prototype (#47)

* Add Stop datatype, and r/w endpoints to Agency

* Add proper validation when POSTing stops

* Add some basic error handling

* Start adding tests...

* Clean up bad db types

* Test cleanup

* More cleanup

* More cleanup pt2

* More cleanup

* Add read back test

* More tests

* Last of the tests?

* Some formatting cleanup

* Remove unused import

* Make geography_id optional for stops, add lat/lng requirement

* Cleanup

* Review nits

* Clean up schema, use SqlVals()

* Remove markdown plugin

* Update carshare -> car, and add moped type to vehicleTypesCountMapSchema definition

* Break build... this is intentional, I promise

* Revert "Break build... this is intentional, I promise"

This reverts commit d8ba98963e3d2c523aa8212e9715f3adf76bd057.

* Fixture updates

* Add recorded column

* Cleanup

* Add option to skip bbox check when fetching vehicles by bbox (#113)

* [mds-audit] Speed up VIN lookup by parallelizing cache reads (#110)

* Speed up VIN lookup by parallelizing cache reads

* Fix build error (#114)

* Update VS Code ESLint settings (#117)

* [mds-db] Refactor makeReadOnlyQuery to permit use of SqlVals() (#116)

* Refactor makeReadOnlyQuery to permit use of SqlVals()

* Fix minor oversight

* [mds-agency] [mds-web-sockets] Basic outbound websocket implementation for MDS-Agency (#99)

* First pass at WS

* Add tsconfig

* Add basic websocket emission to mds-agency

* Emission cleanup

* Add tsconfig

* Cleanup

* Dependency cleanup

* More dependency cleanup

* WIP

* More WIP

* WIP

* WIP

* Manual event triggering works, wheee

* Remove agency socket interaction, to be replaced with reading from KNE

* Added basic auth handling, need to add logic for token expiration. Could also just add a 30s heartbeat requirement.

* Validate auth

* More auth cleanup

* Header validation cleanup

* Generate random events and telemetry to send down websocket

* Nits

* Add outbound ws client

* Message emission working

* Purge console.log

* Clean up authorizers

* Cleanup

* Bump deps

* Bump deps

* Cleanup

* Test coverage

* Cleanup

* Moar cleanup

* So much cleanup

* Bap

* nits

* Moar nits

* Refactor async in forEach to use Promise.all()

* Rename nit

* Export cleanup

* Exception handling cleanup

* Add missing index.ts file

* Remove stale clients

* Make sure to kill the connections too

* Exec function

* Minor authorizer refactor (#111)

* Force exit for agency tests to close open socket

* Moar exit forces

* Import alignment

* Cleanup

* Changes for internal routing and basic helm config

* Remove test code

* Cleanup

* [mds-agency] Fix import of package with no default export (#120)

* [mds-web-sockets] Check for admin:all scope when authenticating clients (#121)

* Add admin-all scope check when authenticating ws clients

* Nullish coalesce all the things

* Remove unnecessary check

* Fix image name for mds-web-sockets (#122)

* .Values.api.env will populate all APIs' environments (#92)

* [mds-web-sockets] Don't always start WebSocket server (#123)

* Don't always start WebSocket server

* Add missing launch file

* use branch name for docker image tag suffix (#124)

* use literal branch name as docker suffix

* portable sed

* [mds-web-sockets] Fix PING message handling

* [mds-processors] Refined Processor Implementation (#104)

* Cleanup

* update requirement versions

* revert to original dir copy

* cleanup

* Beginning of cleanups

* more cleanup/ removed streaming logic scoped for Q1

* trip-proc cleanup

* type fixes

* ts refactor

* helm reconcile

* mds-logger

* metric cleanup

* helm fixes

* breaking db methods into seperate file

* cache/db refactor

* Clean up geo.ts

* Clean up annotation.ts

* Clean up proc-event.ts

* Clean up proc.ts

* Clean up misc

* Clean up metrics.ts

* Update MetricsTableRow type

* Resolve/suppress all linter warnings, and resolve compilation errors.

* Add eslint-reasons to most directive disables

* Refactor geo.ts files into utils

* Add stricter typing

* Cleanup

* Fix dependencies/versions

* cache/db fixes

* proc fixes

* mdsctl: cleanup (comment out nats)

* Cleanup

* Optional chaining

* Add ParseError

* upgrade natss

* mdsctl: update

* mdsctl: update bin/mdsctl add knative(broker,channel)

* fix typo

* mdsclt: ko hack, mds pause for natss

* mdsctl: s/pause/condition/g

* [WIP] [mds-metrics] Add dump all metrics rows API endpoint (#94)

* WIP

codify types

add initial request handler test

add query construction tests

named args obj (no more defaults)

Add explanatory comments

more comments

* s/timestamp/start_time

* rename some things + cleanup

* update test

* fix incorrect import

* fix type signature

* mdsctl: kne w/o namespace labeling

* Remove default param val

* - Add id column to reports tables
- Restore sorting of code

* Do not initialize the db at startup!

* Reduce unnecessary switch cases

* Fix lint error that's been breaking build

* Upgrade to cloudevents 1.0 with typings

* added presets to mdsctl

* update proc-event

* mdsctl uncomment natss

* Remove deprecated Redis stream

* Remove package-lock file

* Generic server for handling knative cloudevents

* Make handler async

* Unit test placeholder

* [mds-metrics] Adds geography_id and provider_id query params to /metr… (#98)

* [mds-metrics] Adds geography_id and provider_id query params to /metrics/all

Usage: GET /metrics/all?geography_id={}&provider_id={}

Both fields are optional.
Each id query param must be a valid UUID or you'll get a 500.

TODO test w/ cURL or Insomnia @evanxadkins

This is also going into PR #88

Closes [MDSAAS-487](https://lacunadotai.atlassian.net/browse/MDSAAS-487)

* fix up lint and add comment

* remove geography_id query param

* initial stab at vehicle_type querying

* vehicle_type query param support

* add english bin_size support

* switch to nifty time utils

* crazy time validation logic

* add note

* fix mutable moment bug

* add note

* clean up request handler

* clean up notes

* add tsv format query param

* add note

* add some tsv export tests

* fix failing test

* tests should be more stable now

* fix import

* purge link

* remove flaky test (again)

* lint

* type enforcement

* eslint errors

* Regenerate lock file

* Switch trip/provider handlers to event server

* Clean up a couple test commands/timeouts

* Add better parsing logs

* trip-proc update

* promise handling

* prov-proc update

* metrics update

* Handle parse errors from cron source events

* Improve test coverage

* Minor cleanup

* Feature/jwtodd events trips provider processor (#101)

* nats: internal deployment of processors

* cjs: remove hardwired minute interval

* helm: fix cronjobsource schedule

* processors: add cronjob template

* interval: 1m

* event sync

* nats: processor (internal-apis; no need to port-forward), simulator cli, natss broker/channel

* helm/mds: template cleanup(broker, processor-cronjob)

* mdsctl: tad better ko ref

* helm/mds cleanup

* mdsctl: make natss a pre-condition of mds

* mdsctl: cleanup

* bug fixes

* Package update

* [mds-metrics] Add stub endpoint (#103)

* add stub endpoint

* fix async/string

* simplify JSON tsv processing

* re-enable/fix up tsv export in main api

* WIP

* cache update

* cache update

* Update test scripts for new packages

* Update versions of mds-core dependencies

* Add missing dependencies, fix build/test errors

* package updates

* minor syntax changes

* mdsctl: sed of osx; todo: fix for all

* mdsctl: better (osx,windows) natss install

* multimodal changes

* mdsctl: support deleting 2+ images

* mdsctl: build image cleanup, simulator build

* event_count metric addition

* [mds-db] [mds-metrics] Add SqlVals() to processors db queries (#107)

* refactor to use SqlVals()

* revert unrelated changes

* helm/mds: +mds-config, for t in helm/mds/templates/* s/key/name/g, move natss-channel into deploy namespace (eg mds); verified: simulator, logs mds-(event,provider,trip)-processor

* Add unit tests to mds-event-processor (#112)

* Revert "[mds-db] [mds-metrics] Add SqlVals() to processors db queries (#107)"

This reverts commit 0cee743787328d6ffa4bbca963370a3030218f4d.

* fix weird lint error

* initial test

* crlf yikes

* moar tests

* lower codecov

* more unit tests

* small helm changes

* sqlVals

* small error handling changes

* Revert "sqlVals"

This reverts commit f2299b25606045e219fe0a1c374ae8140afbd89b.

* vscode settings

* kne/natss:0.11.0, multi-tenant nats/redis msg routing (#115)

* mdsctl: delete 2+ images

* filter prep

* natss: conditionally prefix the event:type with the tenantId else use 'mds' as the default

* rm tenantId dbug

* events/multi-tenancy: patch db statments to vary event-type prefix, attempted to push the tenentId down to utils but bad things happened to my system; next: test concurrent tenants, verify db statements

* kne: move deploy process to helm (from kubectl apply -f [url])

* nats: helm'ify

* nats/helm: pin to gcr

* nats: prioritize (un)install natss via helm

* redis: multi-tenant

* multi-tenant: cleanup

* mdsctl: not to sync kne-contrib from a release

* redis/multi-tenant: clean up comment

* redis/multi-tenant: clean up argument quoting

* mdsctl: helm kne-eventing-init

* nats/multi-tenant: filter out tenant-routing prefix prior to forwarding the message to the handler

* natss event processor: clean-up

* helm/kne-contrib: r0.11.0 natss-channel.yaml

* mdsctl: fixme cleanup

* tenantid: better ternary

* Clean up cloud event parsing.

* Suppress deprecation warnings

* Feature/maxj/sqlvals (#119)

* [mds-audit] Speed up VIN lookup by parallelizing cache reads (#110)

* Speed up VIN lookup by parallelizing cache reads

* Fix build error (#114)

* Update VS Code ESLint settings (#117)

* [mds-db] Refactor makeReadOnlyQuery to permit use of SqlVals() (#116)

* Refactor makeReadOnlyQuery to permit use of SqlVals()

* Fix minor oversight

* re-enable sqlvals

* [mds-agency] [mds-web-sockets] Basic outbound websocket implementation for MDS-Agency (#99)

* First pass at WS

* Add tsconfig

* Add basic websocket emission to mds-agency

* Emission cleanup

* Add tsconfig

* Cleanup

* Dependency cleanup

* More dependency cleanup

* WIP

* More WIP

* WIP

* WIP

* Manual event triggering works, wheee

* Remove agency socket interaction, to be replaced with reading from KNE

* Added basic auth handling, need to add logic for token expiration. Could also just add a 30s heartbeat requirement.

* Validate auth

* More auth cleanup

* Header validation cleanup

* Generate random events and telemetry to send down websocket

* Nits

* Add outbound ws client

* Message emission working

* Purge console.log

* Clean up authorizers

* Cleanup

* Bump deps

* Bump deps

* Cleanup

* Test coverage

* Cleanup

* Moar cleanup

* So much cleanup

* Bap

* nits

* Moar nits

* Refactor async in forEach to use Promise.all()

* Rename nit

* Export cleanup

* Exception handling cleanup

* Add missing index.ts file

* Remove stale clients

* Make sure to kill the connections too

* Exec function

* Minor authorizer refactor (#111)

* Force exit for agency tests to close open socket

* Moar exit forces

* Import alignment

* Cleanup

* Changes for internal routing and basic helm config

* Remove test code

* Cleanup

* [mds-agency] Fix import of package with no default export (#120)

* remove quotes around vals.add()

* follow existing pattern for IN

* clean up unused imports

* more cleanup

* more lint?

* helm/mds: fix values(apis,processors)

* Fix Cloud Event emitter cache

* Remove deprecated test

* Restore cloud event test

* minor proc package fix

* Feature/maxj/additional query params (#118)

* specify multiple provider_id query values

* clean up types, normalize logic to arrays

* add multiple bin processing support

* clean up bin_size handling

* add some additional handling code

* %s/let/return/g

* clean up types/naming

* clean up default arg

* fix test

* mdsctl: fully helm'ify the mds install (by labelling the namespace; noting the cost is an extra 'default' in-memory broker)

* CE prefix metric query removal

* Dependency updates

* added README

* Small cleanups

* Fix helm template

* Fix project reference

* Add unit test to fix coverage

* add recorded column to reporting tables

* Update README.md

* added now util function

* Update README.md

* run triggers on the top of the hour

* helm/kne: remove in-memory-controller; tested with `mdsctl -p processors install:mds,simulator` and observing events propogated to logs

* helm/mds: add `postgresql.password` empty-by-default field with clarifying comment

* helm/knative: remove imc k8s deploy

* [mds-provider-processor] A few unit tests for mds-provider-processor. (#128)

* A few unit tests for mds-provider-processor.

* Fixed test threshold.

* this is not right but it makes tests pass

* Clean up project references

* Adding cache.initialize for consistency.

* unit tests for metric utils

* formatting

* cleanup

* Restore configmap functionality

* type reduction

* hour cron interval

* Mount config volume in all pods (#130)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>
Co-authored-by: Michael Durling <michael@ellis-and-associates.com>
Co-authored-by: James Todd <99516+jwtodd@users.noreply.github.com>
Co-authored-by: Max Johansen <macsj200@gmail.com>
Co-authored-by: Alex Gottschalk <389982+invertigo@users.noreply.github.com>
Co-authored-by: janedotx <jane.x.huang@gmail.com>

* [mds-web-sockets] Add proper env-injection to mds-web-sockets (#133)

* mdsctl: remove dashboard, get cli:postgresql and cli:redis working again (#126)

* mdsclt: rm dashhboard, fix cli

* mdsctl: get cli:[postgresql|redis] working again

* mdsctl: add helm@2 to PATH given it is installed in a non-default/opt location; note: dep on #104, will work as-is if pushed prior

* helm/mds: match #104 re $.Values.apis.env parse error

* mdsctl: helm init cleanup, don't display usage message when an error is present

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* [mds-agency] Don't retry then send a 500 on a failed cache/stream/socket write (#134)

* Don't retry then send a 500 on a failed cache/stream/socket write

* Use recorded_event rather than event

* Refactor telemetry writing

* fix crash if values not present (#125)

* [mds-audit] Return last active device registered if db has duplicates for provider + vin (#127)

* Return last device registered if db has duplicates for provider + vin

* Fix lint

* Prefer an active device when de-duping

* Load configuration data via service (#132)

* Load configuration data via service

* Rename a few things...

* Fix tests

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* [mds-agency] [mds-utils] Extract normalizeToArray() and leverage in Daily  (#136)

* refactor normalizeToArray()

* use new helper

* [mds-*] Append commit hash to image tag if on non-master branch (#139)

* Append commit hash to image tag if on non-master branch

* Review nits

* replace heisentest with unit tests (#138)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* Update package dependencies (#140)

* [mds-web-sockets] Initial web-socket test to get the framework in place. (#137)

* Initial web-socket test to get the framework in place.

* Minor refactor

* mdsctl: number of usability improvements (#135)

* mdsctl: seed PATH for non-default install bins; confirm remote-deploy only once

* mdsctl: tiller aws

* mdsctl: cleanup

* mdsctl: better remoting, add ability to cycle a pod, lessen remote warning check

* mdsctl: add cycle default, all services and procesors

* mdsctl: cleanup neg responses

* mdsctl: nit

* mdsctl: deploy in-cluster pgcl/redis-cli; note: can remove pcli/redis from brew

* mdsctl: rm brew(pgcli, redis)

* mdsctl: rm dead preset hook

* mdsctl: fix bugs(uninstallHelm, don't require MDS_SIMULATOR_REPOSISORY if MDS_SIMULATOR is set to a viable install)

* mdsctl: move some non-obvious configuration switches to comment-notes

* helm/natss: provide ability to override default resources(requests.cpu, limits.memory)

* readme: update mdsctl invocation

* mdsctl: +grafana; note: not configured correctly just yet

* mdsctl: +(un)install(dnslookup,curl,grafana)

* mdsctl: support kn(eventing,monitoring,serving)

* mdsctl: helm-delete by namespace

* mdsctl: touch ~/.bashrc if it doesn't exist, better bash command-to-brew-recipe mapping support

* helm/natss: set default memory to 512M, override with 32M

* mdsctl: install:knative(monintoring); will be continued in another branch

* natss: default memory=512M, fix mdsctl dns-lookup ns bug

* mdsctl: processors(+websockets)

* [mds-agency] [mds-web-sockets] Improve agency/telemetry error handling, and add basic health check to WebSocket client (#141)

* First pass at cleaning up telemetry error handling

* Patch the client to properly handle not being able to connect to web-sockets

* More attemps at fixing client code...

* Moar cleanup

* More cleanup

* add unified state representation (#144)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* [mds-metrics] Fixes metrics scope processing (#146)

* adds metrics:read and metrics:read:provider scoping

* fix error msg

* add note

* push provider query logic to middleware

* simplify logic

* fix import

* lint

* fix unit tests

* coverage

* alphabet

* add state transition diagram (#148)

* Support async access token scope validators (#147)

* Support async access token scope validators

* Dependency updates

* ESLint/Prettier

* remove self loop (#149)

* [mds-utils] Remove unused state transition function (#145)

* remove unused function

* lint

* add expected transition values

* Make redis multi calls async (#151)

* [mds-*] Upgrade to Node.js 12 (v12.14.1) runtime (#154)

* Upgrade to Node.js 12 (v12.14.1) runtime

* Upgrade node types

* [mds-metrics] Add unauthorized error if query params attempt to exceed scope (#156)

* add unauthorized error if query params attempt to exceed scope

* reorder scope checking

* Feature/eadks proc cleanup (#150)

* cleanup spike

* telemetryMap structure change

* optimize registered count metric

* fixed test case

* null vehicle_type cache read bug fix

* service area test cases

* test case type enforcement

* cleanup

* query fix

* reduce logic

* removed unused import

* added error msg

* removed unused import

* syntax edits

* migration script and sql logs

* Remove provider API from City Services cluster (#159)

* Remove provider API from City Services cluster

* Remove provider API scopes

* fix heisenbug by moving tests to end (#161)

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* Extend daily timeout (#162)

* increase timeout

* timeout env var

* reorder number cast

* make sure we have AWS env vars before we make the call asking for AWS creds (#164)

* two changes to fix heisenbug pt 2 in policy-author (#165)

* [mds-cache][mds-processors] Cache optimization for event and trip processors (#160)

* cleanup spike

* telemetryMap structure change

* optimize registered count metric

* fixed test case

* null vehicle_type cache read bug fix

* service area test cases

* test case type enforcement

* cleanup

* query fix

* restructure spike

* hscan addition

* proc cache changes

* hscan reformat

* hscan bug fix

* reduced cache maintanence logic

* removed unused import

* edit to cache method names

* merge cleanup

* merge cleanup

* more merge cleanup

* comment syntax

* map edits

* reduced logic

* typdef fixme comment

* revert values.yaml

Co-authored-by: Mark Maxham <max@ellis-and-associates.com>
Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* forward parsing error to client (#166)

* Don't mutate datetime (#167)

* no mutation

* refactor modules + add test

* Add moment-based timestamps

* Export getCurrentDate()

Co-authored-by: Neil Goldader <avatarneil@gmail.com>

* Fix clone error on Windows by removing unit test for invalid filename (#168)

* [mds-stream] [mds-event-processor] NATS implementation for mds-event-processor. (#143)

* First pass at nats implementation for mds-event-processor

* Some cleanup

* Works in-cluster with manual IP for NATS.connect args

* Actually await things

* nats: mdsctl(nats), disable kne/natss

* Remove trigger.yaml

* Check env.NATS instead of env.SINK

* mdsctl: natss

* Switch from vanilla nats to nats-streaming. This enables us to sychronously handle subscriptions.

* stan ftw

* add STAN_CLUSTER_ID env var

* WIP

* WIP

* WIP

* WIP

* Re-add event-server wrapper for mds-event-processor

* Revert some prior changes, structure events with CE

* Add credentials support

* stan.creds

* Add STAN_CREDS env var

* Some changes...

* stan.creds mount

* cleanup

* Stop using mds-event-server because it causes problems, working end-to-end in cluster

* Patch

* infer image:tags from remote registry

* +gcloud

* Re-add getRawData()

* Re-add mds-event-server, switch to using nats-streaming 0.3.0 client

* Clean up dependencies

* Update dependencies

* Ack failed messages

* helm(nats,stan)

* stan/cleanup

* +nats-box

* +nats-box, pin cluster-id

* stan cluster id

* add [nats|stan]x which are single-pod no-auth nats-io/k8s deploys

* helm'ify (nats,stan)x

* Cluster handled pod routing

* +stan=statefulset

* +stans:0.16.2

* u/d

* Event-server refactoring to improve readability

* Add getNats() to get a persistent nats client and save it if it falls over

* Add re-connect logic

* Turns out the nats client will attempt to reconnect for us, groovy.

* Enable reconnects

* Misc cleanup

* disable istio-injection in nats deployments

* prune kne

* cleanup

* stan: disable istio

* mdsctl: patch nats helm so that it's operations are distinct from init,cluster

* (un)install:stan fix

* cleanup natsbox conf

* install {tools}/helm:2.14.1 if not preent in the env

* bump stan disk allocation

* allow for more-more-then-one (natsbox, curl, dns) deploys

* add ServiceEntry egress for nats, and DestinationRule to disable mTLS for the destination namespace

* switch default nats namespace to "nats" from "default", use correct nats endpoint for stan

* mdsctl: +(nats,stan)-local, rm(kne*); next: conf mds->nats.nats

* mds: add env NATS=nats.nats, set STAN_CLUSTER=stan; would like to remove env STAN, STAN_CREDENTIALS, SINK

* patch

* helm/mds/.../egress.yaml: fix typo

* patch: get image versions

* +kubectx, helm/mds/deployment conf nats ns

* cold-start install patch

* cleanup

* stan: set max-message,max-bytes=0 (unlimited)

* mdsctl: keep-cache support for yarn

* trip-proc cron job

* Cron restart policy fix

* cronjobs in the house

* sync mds-trip

* cleanup

* mds/templates: image w/ & w/o registry

* cleanup env

* Remove CE_NAME and STAN in favor of NATS

* Update STAN_CLUSTER to nats-streaming

* Switch to using env.NATS for streaming

* More switching to env.NATS

* provider-proc cron edits

* Await parseTripsEvents

* u/d readme(mdsctl)

* u/d mdsctl

* u/d mdsctl

* mdsctl: support provider=aws:imageTag (get latest tag by image name)

* mdsctl: u/d readme

* Optional processor fix

* mount mds-config configmap

* Remove db.startup, cache.startup, getConfig. They're not working properly at the moment

* Remove db.startup from mds-provider-processor

* +surveyor, rename stan-statefulset to nats-streaming-server

* helm/mds: disable (trip, provider) processors, as they are invokved via cron

* surveyor: rm creds

* helm/natsbox: rename to nats-box

* Force exit after function is done

* use namespace name as default TENANT_ID

* Fix bad variable injection

* allow empty tenantId for legacy installs (ladot), use namespace if key is missing from .Values

* helm/mds: clean-up proto dead-code

* not using knative anymore

* consistent naming

* cleanup resource names

* differentiate nats/stan config-map names

* s/example-nats/nats/

* switch cron to hourly, turn off migrations for agency

* Update mdsctl

* Update README.md

* Update mdsctl

* Update mdsctl

* remove dependency on ~/.bashrc

* mdsctl: remove persisting conf in ~/.bashrc

* minor refactor - put event-processor into APIs as a service deployment and put trip- and provider- processors into cronjobs.  retire old metrics-sheet specific cronjob

* pull out nats-surveyor to be applied to a subsequent branch

* Drop nullish coalescence operator in favor of logical OR for TENANT_ID

* Update TENANT_REGEXP

* update template to match yaml, remove spurious env vars

* helm/nats: patch stan conf (store typo)

* Switch processors -> cronjobs

* mdsctl: build/yarn-clean

* helm: prune nats-local

* Typo

* Env var defaults cleanup

* u/d readme re k8s conf

* Add comment detailing writeCloudEvent

Co-authored-by: James Todd <99516+jwtodd@users.noreply.github.com>
Co-authored-by: Alex Gottschalk <389982+invertigo@users.noreply.github.com>
Co-authored-by: Evan Adkins <53316916+evanxadkins@users.noreply.github.com>

* Update README.md

readme/docker-desktop-config cleanup

* Assume cache as source-of-truth for  output to get time since last event due to performance problems in production (#169)

* istio-disable cronjobs (#170)

* [mds-*] Add Okteto Configuration for in-cluster debugging (#171)

* First-pass at okteto integration

* Add per-service okteto config

* Move config back to top level

* Update docs, fix missing name

* update docs

* Oops

* Update docs

* Update docs moar

* Docs updates

* Bump okteto resources massively to avoid crashes at debug init (#172)

* Make all tests async (#173)

* change envoy idle timeout to offset from nodejs idle timeout.  reduces incidents of a race condition where envoy attempts to connect with a timed-out node, resulting in a 503 UC

* [mds-db] Refactor en-mass import/exports in mds-db to be more readable (#177)

* Update Config API to support partial configs (#178)

* Refactor config service to improve error handling and test coverage

* Add partial config support

* Unit tests for partial config

* Change error (404) response body

* Upgrade dependencies; fix typescript/lint errors (#179)

* Export interfaces (#181)

* move annotation for preventing istio sidecar injection to correct met… (#180)

* move annotation for preventing istio sidecar injection to correct metadata location

* move annotation for preventing istio sidecar injection to correct metadata location

* remove dupe

* too many specs and templates

* [mds-web-sockets] First pass at some internal JWT validation. (#157)

* Promisify all the things

* Cleanup

* Working with Bearer tokens

* WIP

* Working tests whoop

* Fix error message

* Increase default timeout values to reduce spurious 503 errors from Istio (#184)

* Update Agency, Policy, and the root README docs (#182)

* OMF-26 API Version Middleware (#189)

* API Version Middleware

* Minor corrections

* Swallow STAN fatal errors. Previously, these would cause a node crash dump. (#190)

* Convert empty event.event_type_reason to null (#195)

* [mds-cache] Clean up promise logic in hreads (#153)

* Await batch hgetallasync rather than awaiting each invocation

* Remove Promise junk

* Remove janky promises, update typedef

* Promise.all all of the things

* Remove inner Promise.all

* Linterz

* yarn upgrade-interactive --latest (#193)

* Fix redis multi definition (#196)

* Jurisdiction API (GET, POST endpoints) (#187)

* getOne/getAll jurisdiction service methods

* create Jurisdiction service method

* initialize/shutdown jurisdiction service

* Initial GET /jurisdictions endpoint

* Drop schema after unit tests

* Clean up return statement

* Add container image for MDS Jurisdiction API

* API enpoints for create / find jurisdiction

* Add validation on create

* Update copyright notices

* Use port 4011 for local jurisdiction API

* Return 404 when jurisdiction not found

* Add some error typing to jurisdiction service

* Remove unused type

* Scopes for writing jurisdictions + api tests

* Add scopes for jurisdiction read

* Add jurisdiction claim filtering

* Stop tests from failing in the presence of extra db tables

* mdsctl installs mochawesome; commiting

* Rename a scope

* Add copyright notices

* Add versioning middleware to Jurisdiction API

* Updated lock file

* Merge origin/develop and update dependencies

* mdsctl yarn.lock updates

* Add some DB connection logging

* Use R/W connection for running TypeORM migrations (#197)

* Clean up Versioned API types; Reorganize Jurisdiction/Native APIs (#198)

* Refactor Jurisdiction API

* Clean up Native API types

* Rename some private interfaces/types

* Use array destructuring

* Reorganize Native API middleware/handlers

* Rename utils files

* Catch db connection errors separately (#199)

* [mds-provider-processor] Provider filter for device cache  (#174)

* provider-device cache call

* cron schedule revert

* updated tests

* test case bug

* cleanup

* nullish coalescing

* registered device map refactor

* registered device map refactor

* cleanup

* [mds-web-sockets] Add structured response for authentication success and failure (#204)

* Add structured response for authentication success and failure

* Fix test

* [mds-agency] [mds-utils] Add better object validation to agency. (#200)

* Add device object validation, begin working on building out event validators

* Add event and telemetry validation to agency request handlers

* Remove comment

* Refactor to match other usage better, add allowUnknown validator option

* More cleanup

* Cleanup

* Endpoints to update/delete a jurisdiction (#201)

* Update a single jurisdiction

* Add unit tests

* Sort versions by timestamp

* Support "deleting" jurisdictions (with tests)

* Separate out ORM operations into their own module.

* Fix import statements

* Tidy up some error messaging

* Add a comment

* carve out exception to jwt auth for websockets api (auth is done internally due to different headers for ws protocol) (#192)

* [mds-*] Merge in OMF/develop (#206)

* Kubernetes Contribution (#233)

* mdsctl/preset: onward to gh

* mdsctl: landed in gh

* Add additional test + simplify sum logic

* generalize over summing any columns

* fix test index type

* Split up mds-db into more manageable files.

* mdsctl: open:dashboard patch

* Increase test timeout to support k8s DB/Redis (#4)

* mdsctl: uinstall:simulator

* mdsctl: add preset:no-events

* mdsctl: -p:no-events

* mdsctl: install:simulator python pathing

* review feedback, boiling all of index.ts down

* k8s/util-namespace: bring forward from cola

* Add last provider event to audit_events table on every audit event

* Add endpoint to get vehicle by provider and vin

* mdsctl: support running from path

* Add geos summaries endpoint.

* mdsctl: support running from any directory, add --quiet option to be 'less verbose' (can likely do more)

* Consolidate provider event flattening, fix function-called-before-definition lint error

* review feedback

* mdsctl: provide a means to disable (postgresql, redis) persistence

* mdsctl: conditionally update ~/[rc] to add mdsctl to PATH env

* mdsctl: conditionally add mdsctl to PATH

* mdsctl: fix install:logging namespace

* Bump versions

* mdsctl: override fluentbit

* mdsctl: logging local

* mdsctl: don't istio sidecar logging

* test fails fuck

* mdsctl: push logging to util namespace

* Fixed Geography type.

* Using fancier types.

* improving tests

* review feedback

* mdsctl: leverage getopts/optspec

* mdsctl: cleanup

* Resolve Geography related build errors.

* Implement count minimums.

* Break readGeographies into separate methods for reading Geography and GeographySummary objects.

* Rename readGeographiesSummary -> readGeographiesSummaries, and replace duplicate logic with a call to readGeographies then simply strip out the geography_json

* Minor tweaks due to vscode doing some poor automagical renaming

* Better parameter destructuring

* Rename readGeographiesSummaries -> readGeographySummaries

* Resolve lint error, oops

* Add additional logging info to getProviderMetrics()

* Bump version

* mdsctl: tab completion

* [mds-agency] s/car/carshare/g MDSAAS-277

Slack discussion https://lacuna-tech.slack.com/archives/CHXQXQK5M/p1571325224046400

* update ref to enum key

* add a small test

* mdsctl: remove python env dep

* mdsctl: s/presets/preset/g

* Resolve bad import

* finitio

* MDSAAS-275 Remove AWS Lambda handlers, etc. (#25)

* MDSAAS-275 Remove AWS Lambda handlers, etc.

* Remove lambda deployment script runner

* Run ESLint concurrently before unit tests

* Remove AWS reference from comment

* Remove explicit typing

* Bump version

* Publish

 - @container-images/env-inject@0.1.13
 - @container-images/mds-agency@0.1.14
 - @container-images/mds-audit@0.1.15
 - @container-images/mds-compliance@0.1.14
 - @container-images/mds-daily@0.1.14
 - @container-images/mds-metrics-sheet@0.1.15
 - @container-images/mds-native@0.0.12
 - @container-images/mds-policy-author@0.1.14
 - @container-images/mds-policy@0.1.14
 - @container-images/mds-provider@0.1.14
 - @mds-core/mds-agency@0.0.16
 - @mds-core/mds-api-authorizer@0.1.14
 - @mds-core/mds-api-helpers@0.1.14
 - @mds-core/mds-api-server@0.1.14
 - @mds-core/mds-audit@0.1.26
 - @mds-core/mds-cache@0.1.14
 - @mds-core/mds-compliance@0.1.15
 - @mds-core/mds-daily@0.0.15
 - @mds-core/mds-db@0.1.14
 - @mds-core/mds-logger@0.1.12
 - @mds-core/mds-metrics-sheet@0.0.15
 - @mds-core/mds-native@0.0.12
 - @mds-core/mds-policy-author@0.0.11
 - @mds-core/mds-policy@0.0.15
 - @mds-core/mds-provider@1.0.14
 - @mds-core/mds-providers@0.1.14
 - @mds-core/mds-stream@0.1.14
 - @mds-core/mds-test-data@0.1.14
 - @mds-core/mds-types@0.1.11
 - @mds-core/mds-utils@0.1.14

* comments

* Rename previous_geographies column and update deleteGeography so it handles all cases where read_only is not true.

* Fix bad null checking. Closes MDSAAS-289

* MDSAAS-202 Upgrade to Node.js 10.16.3 LTS runtime (#12)

* MDSAAS-202 Upgrade to Node.js 10.16.3 LTS

* Suppress node deprecation warnings

* review feedback

* bump root version

* Publish

 - @container-images/env-inject@0.1.14
 - @container-images/mds-agency@0.1.15
 - @container-images/mds-audit@0.1.16
 - @container-images/mds-compliance@0.1.15
 - @container-images/mds-daily@0.1.15
 - @container-images/mds-metrics-sheet@0.1.16
 - @container-images/mds-native@0.0.13
 - @container-images/mds-policy-author@0.1.15
 - @container-images/mds-policy@0.1.15
 - @container-images/mds-provider@0.1.15
 - @mds-core/mds-agency@0.0.17
 - @mds-core/mds-api-authorizer@0.1.15
 - @mds-core/mds-api-helpers@0.1.15
 - @mds-core/mds-api-server@0.1.15
 - @mds-core/mds-audit@0.1.27
 - @mds-core/mds-cache@0.1.15
 - @mds-core/mds-compliance@0.1.16
 - @mds-core/mds-daily@0.0.16
 - @mds-core/mds-db@0.1.15
 - @mds-core/mds-logger@0.1.13
 - @mds-core/mds-metrics-sheet@0.0.16
 - @mds-core/mds-native@0.0.13
 - @mds-core/mds-policy-author@0.0.12
 - @mds-core/mds-policy@0.0.16
 - @mds-core/mds-provider@1.0.15
 - @mds-core/mds-providers@0.1.15
 - @mds-core/mds-stream@0.1.15
 - @mds-core/mds-test-data@0.1.15
 - @mds-core/mds-types@0.1.12
 - @mds-core/mds-utils@0.1.15

* match path prefix, optionally followed by a slash, optionally followed by [-a-z] character set

* Populate publish_date on geographies.

* review feedback

* incrase timeout

* Fix bad geographies migration

* Minor fix

* Bump main version

* Publish

 - @container-images/env-inject@0.1.15
 - @container-images/mds-agency@0.1.16
 - @container-images/mds-audit@0.1.17
 - @container-images/mds-compliance@0.1.16
 - @container-images/mds-daily@0.1.16
 - @container-images/mds-metrics-sheet@0.1.17
 - @container-images/mds-native@0.0.14
 - @container-images/mds-policy-author@0.1.16
 - @container-images/mds-policy@0.1.16
 - @container-images/mds-provider@0.1.16
 - @mds-core/mds-agency@0.0.18
 - @mds-core/mds-api-authorizer@0.1.16
 - @mds-core/mds-api-helpers@0.1.16
 - @mds-core/mds-api-server@0.1.16
 - @mds-core/mds-audit@0.1.28
 - @mds-core/mds-cache@0.1.16
 - @mds-core/mds-compliance@0.1.17
 - @mds-core/mds-daily@0.0.17
 - @mds-core/mds-db@0.1.16
 - @mds-core/mds-logger@0.1.14
 - @mds-core/mds-metrics-sheet@0.0.17
 - @mds-core/mds-native@0.0.14
 - @mds-core/mds-policy-author@0.0.13
 - @mds-core/mds-policy@0.0.17
 - @mds-core/mds-provider@1.0.16
 - @mds-core/mds-providers@0.1.16
 - @mds-core/mds-stream@0.1.16
 - @mds-core/mds-test-data@0.1.16
 - @mds-core/mds-types@0.1.13
 - @mds-core/mds-utils@0.1.16

* better regex - match the pathPrefix followed by either EoL or a slash and zero or more characters

* route by FQDN in ingress

* update values for ingress routing by domain

* ALTER TABLE RENAME COLUMN ... TO ... (#36)

* remove letsencrypt option - never worked reliably and not the ingress is not compatible with non-LE TLS (not possible to switch back and forth)

* remove letsencrypt option - never worked reliably and not the ingress is not compatible with non-LE TLS (not possible to switch back and forth)

* standardize API TCP ports to 4000, remove some never-used selectors for enabling/disabling DB and Cache env vars, make Eventing selector global instead of per-api

* bump package version

* Publish

 - @container-images/env-inject@0.1.16
 - @container-images/mds-agency@0.1.17
 - @container-images/mds-audit@0.1.18
 - @container-images/mds-compliance@0.1.17
 - @container-images/mds-daily@0.1.17
 - @container-images/mds-metrics-sheet@0.1.18
 - @container-images/mds-native@0.0.15
 - @container-images/mds-policy-author@0.1.17
 - @container-images/mds-policy@0.1.17
 - @container-images/mds-provider@0.1.17
 - @mds-core/mds-agency@0.0.19
 - @mds-core/mds-api-authorizer@0.1.17
 - @mds-core/mds-api-helpers@0.1.17
 - @mds-core/mds-api-server@0.1.17
 - @mds-core/mds-audit@0.1.29
 - @mds-core/mds-cache@0.1.17
 - @mds-core/mds-compliance@0.1.18
 - @mds-core/mds-daily@0.0.18
 - @mds-core/mds-db@0.1.17
 - @mds-core/mds-logger@0.1.15
 - @mds-core/mds-metrics-sheet@0.0.18
 - @mds-core/mds-native@0.0.15
 - @mds-core/mds-policy-author@0.0.14
 - @mds-core/mds-policy@0.0.18
 - @mds-core/mds-provider@1.0.17
 - @mds-core/mds-providers@0.1.17
 - @mds-core/mds-stream@0.1.17
 - @mds-core/mds-test-data@0.1.17
 - @mds-core/mds-types@0.1.14
 - @mds-core/mds-utils@0.1.17

* Upgrade dependencies (#29)

* Upgrade dependencies to latest versions

* More upgrades

* hard-code container port

* initial commit, docker-compose with an nginx gateway proxy

* default ports

* move out of subdir

* simplify PG_HOST_READER - can leave unset and the mds-db code will fall back to PG_HOST, make PG_PORT optional

* add optional slack integration vars, common secret for APIs

* changed secret name

* Add standard API request logging (#43)

* Add standard request logging

* port must be string not int

* fix slack token and secret

* fix value for ingress domain FQDN

* Remove provider event processor (#41)

* Remove provider event processor

* Drop deprecated provider tables

* Enable SQL logging during migrations

* Only drop tables that are/were part of the schema

* need to quote domains in case of a wildcard globa ("*")

* Remove read_only column from geographies.

* Remove unused provider-stats cache methods

* bump version

* hard-code port number

* correct secret name

* template docker-compose with latest build versions

* Add perf logging to MDS Daily db/cache calls

* fluentbit (mds/requirements rm, util/values pin to es host) (#50)

* narrow fluentbit refs

* fluentbit: rm from mds/requirements, pin to elasticsearch-master in util/values

* helm/(mds,util): rm fluent from mds

* pull changes from fix/michael/remove-provider-api

* Add perf logging to MDS Daily db/cache calls

* Resolve linter errors

* Bump root package version

* Publish

 - @container-images/env-inject@0.1.17
 - @container-images/mds-agency@0.1.18
 - @container-images/mds-audit@0.1.19
 - @container-images/mds-compliance@0.1.18
 - @container-images/mds-daily@0.1.18
 - @container-images/mds-metrics-sheet@0.1.19
 - @container-images/mds-native@0.0.16
 - @container-images/mds-policy-author@0.1.18
 - @container-images/mds-policy@0.1.18
 - @container-images/mds-provider@0.1.18
 - @mds-core/mds-agency@0.0.20
 - @mds-core/mds-api-authorizer@0.1.18
 - @mds-core/mds-api-helpers@0.1.18
 - @mds-core/mds-api-server@0.1.18
 - @mds-core/mds-audit@0.1.30
 - @mds-core/mds-cache@0.1.18
 - @mds-core/mds-compliance@0.1.19
 - @mds-core/mds-daily@0.0.19
 - @mds-core/mds-db@0.1.18
 - @mds-core/mds-logger@0.1.16
 - @mds-core/mds-metrics-sheet@0.0.19
 - @mds-core/mds-native@0.0.16
 - @mds-core/mds-policy-author@0.0.15
 - @mds-core/mds-policy@0.0.19
 - @mds-core/mds-provider@1.0.18
 - @mds-core/mds-providers@0.1.18
 - @mds-core/mds-stream@0.1.18
 - @mds-core/mds-test-data@0.1.18
 - @mds-core/mds-types@0.1.15
 - @mds-core/mds-utils@0.1.18

* Purge mds-provider implementation

* Update LICENSE

* Remove mds-cache weird duplicate functions

* Clean up mds-audit weirdness

* Clean up mds-db weirdness

* Clean up mds-types weirdness

* Clean up helm template weirdness

Co-authored-by: Hunter Owens <owens.hunter@gmail.com>

* Fix yarn.lock

* [mds-policy-author][mds-geography-author] Extract geography author service from policy author. (#203)

* Extract geography author service from policy author.

* review feedback

* review feedback

* review feedback

* RF

* RF

* satisfy linter

* SqlVals and vals_list used to sanitize db input.

* rf

* helm unittest (#209)

* mdsctl: helm unittest service

* heml uuniest: auth

* helm unittest: auth disable

* helm unittest: autoscaler

* helm unittest: +configmap

* helm unittest: cronjob

* helm unittest: egress

* helm unittest: ingress

* helm unittest: ingress domain

* helm unittest: deployment

* helm unittest: deployment secret

* helm unittest: deployment overrides

* helm unittest: better suite labels

* helm force upgrade

* mdsctl: ensure resolvable k8s endpoint (thx jane)

* Reorder writing an event and related telemetry to avoid race condition where event is written but telemetry has not been written yet (#205)

* [mds-agency] [mds-web-sockets] Add NATS-Streaming consumption for mds-web-sockets (#208)

* Add mds-web-sockets consumption of NATS stream rather than agency having to persist a direct connection with the web-socket server

* Clean up okteto.yml

* Handle STAN errors better

* eslint-ignore

* Istanbul, not eslint

* [mds-policy] [mds-geography] Extract read-only geography service from policy service (#211)

* Extract read-only geo points from Policy.

* making mds-policy tests pass

* Tests for mds-geography.

* pacifying linter

* remove todo comment

* removing TODO comment

* Feature/alex/monitoring alerting (#213)

* initial commit, canned alerts for prometheus

* whitespace, add additional alerts

* clean up summaries and descriptions

* clean up slack notifications

* better tracking of container restarts (count over time instead of absolute), better labeling of statefulset failures

* use rate over last five minutes

* bump version

* update lockfile, try removing auth from grafana (thanks james) (currently broken)

* add prometheus datasource, some useful dashboards.  remove login since access requires authenticated kubectl

* update

* ignore downloaded charts

* more meaningful restart rate

* tuning

* add alarms for pod CPU and memory going over limit

* switch disk usage alerts from static to predictive

* Upgrade package dependencies (#212)

* Upgrade package dependencies

* Add type annotation to fix build error

* This test was destined to fail starting yesterday (#215)

* [mds-processors][mds-metrics] removal for OMF release (#210)

* removal spike

* db/cache util and type removal

* turf deps

* purge metrics + more processors

* remove cron from mdsctl

Co-authored-by: Max Johansen <macsj200@gmail.com>

* [mds-policy-author] Minor changes to policy to bring it back in line with the spec. (#217)

* Fixing parameters for policy author.

* Adding User Rule type to Policy.

* [mds-stream] Add outbound kafka stream support (#216)

* Add outbound kafka stream support

* Cleanup

* Review cleanup pt1

* Refactoring and cleanup

* Call KafkaStream.initialize() from top-level initialize method

* Call stream.initialize() at the top of mds-agency/request-handlers

* Update README.md

* Remove stream.initialize() from agency tests

* Remove stream.initialize() from compliance tests

* Writer refactor, and reader implementation

* Fix unhappy exports

* Remove callback fallback :P

* clean up stream init

* Run with new Prettier 2.0 defaults (#221)

* Run with new prettier defaults

* Revert trailingComma to previous default

* Revert arrowParens to previous default

* Removing Slack client and pushover notifications (#226)

* Removing Slack client and pushover notifications

* Implement new method to redact sensitive text

* Kafka Stream Processor (#224)

* Beginnings of initial refactor

* More cleanup of kafkajs, removal of node-rdkafka

* Rename and refactor

* Use Nullable<T>

* Remove temporary type declaration

* Remove file

* Make producers topic specific

* Basic functional streap labeler

* Remove EventServer

* Remove mds-event-server

* Refactor stream processing

* Fix module export issues

* Add container image for vehicle event processor

* Fix package name

* Add helm deployment for event-processor

* await NATS write instead of returning

* Remove reliance on deprecated service_area_id

* Incorporate review feedback

* Merge remote-tracking branch 'origin/develop' into feature/neil/begin-kafka-refactor-to-kafkajs

Co-authored-by: Neil Goldader <neilgoldader@gmail.com>

* [mds-utils] Update relative time parsing to snap to the nearest hour when 'now' is provided as a query param (#223)

* Update relative time parsing to snap to the nearest hour when  is provided as a query param

* Undo old prettier weirdness

* Add caching of device labels (#229)

* Add caching of device labels

* Rename some consts

* Add control methods to stream processor (#232)

* Add control methods to stream processor

* Minor package refactoring

* Add copyright notices

* Remove some spammy logs from mds-cache

* [mds-stream-processor] Add geo labeling for mds-stream-processor (#231)

* Add geo labeling for mds-stream-processor

* Add tests

* Add Sinon devDependency

* Add LatencyLabeler tests

* Filter based on point in shape, then map to geography_id

* Remove now unused import

* Only compute for published geographies

* [ladot-service-areas] [mds-agency] [mds-daily] Purge service-areas (#219)

* Purge service-areas

* Rename mds-test-data/ladot-service-areas to mds-test-data/test-areas

* Get rid of require statements

* Remove mds-metrics-sheet vehicle counts (#228)

* Remove mds-metrics-sheet vehicle counts

* Remove vehicle-counts.ts

* Remove duplicate Nullable<T> definition

* Fix mds-metrics-sheet container image build (#233)

* Reorganize files

* Remove extraneous vehicle-counts references

Co-authored-by: Neil Goldader <neilgoldader@gmail.com>

* Fix event type for vehicle reads in case of a cache miss (#225)

* Fix Project References (#234)

* Agency should not be writing telemetry to socket

* Fix typescript project references

* Make sinon a dev dependency

* Upgrade dependencies

* [mds-native] Removing mds-native. (#214)

* Removing mds-native.

* Removing helm tests.

* Removing reference in nginx.

* [mds-db] [mds-policy-author][mds-compliance] Fix/final separation of policy from geo (#218)

* Publishing a policy no longer publishes a geography. Instead it throws an error.

* minor fixes

* Removing container-images directory for mds-native. (#236)

* Remove leftover artifacts of mds-metrics (#238)

* [mds-stream] [mds-stream-processor] TENANT_ID prefix for Kafka Streams (#235)

* Add TENANT_ID prefix to kafka to support multi-tenancy on one Kafka cluster

* Yarn.lock changes I guess?

* Add a couple source/sink examples (#237)

* Re-add stream init that was removed, thanx git (#239)

* Add FileSource and FileSync connectors for mds-stream-processors (#240)

* [mds-stream-processor] First pass at vehicle telemetry processor (#241)

* First pass at vehicle telemetry processor

* Add getEnvVar method to mds-utils

* Refactor getEnvVar

* Refactor TelemetryFlattener to TelemetryLabeler, add OptionalTelemetryLabeler

* Rename FlattenedTelemetry, update NullableProperties type

* [helm] Update helm for telemetry processor (#245)

* Update helm for telemetry processor

* do not create auth policies or ingress virtualservices for APIs with no pathPrefix (ie internal-only services)

Co-authored-by: Alex Gottschalk <alex.gottschalk@gmail.com>

* Initial Metrics Service (#244)

* Add ORM model for metrics

* Update dependencies

* Create some service helper methods

* Implement metrics filtering on read

* Allow specifying one of more filter values

* Refactor metric query tests to minimize redundancy

* Refactor utility function and add missing licenses

* Remove OptionalTelemetryLabeler

* Code review revisions

* was missing geography api (#242)

* Split metrics-service into server/client modules (#246)

* [mds-stream] Switch from NATS Streaming -> Vanilla NATS (#243)

* First pass

* Minor fix that will be replaced once I merge in

* Retry forever

* waitOnFirstConnect fix

* Fix variable names

* Add typing for NATS messages

* NATS env var refactor

* nats-streaming folder -> nats folder

* Thanks F2

* Path fix

* Enable ES2019 features in node:12.14.1-alpine (#247)

* Refactor ORM Connection Configuration (#249)

* Fix coverage excludes

* Support named connections

* Curry the ORM connection factory functions

* Refactor Jurisdiction Service (#250)

* ServiceProvider interface

* CreateRepository function

* CreateRepositoryMethod function

* Move jurisdiction types/utils to service package (#251)

* Move jurisdiction types/utils to service package

* Use helper method for filter

* Move schema to separate file

* Export default connection for TypeORM CLI (#252)

* Export default connection for TypeORM CLI

* More robust destructuring

* Fix import statement

* Clean up error handling

* Improve test coverage

* Use recommended typescript config for istanbul (#255)

* Use recommended typescript config for istanbul

* Clean up some config files

* Create types for mapping between entity/domain models (#257)

* Create types for mapping between entity/domain

* Fix test failure

* Fluent model mapper syntax

* Use object schema for jurisdiction validation

* Fix imports and make mapper syntax consistent (#258)

* [mds-stream-processor] First pass at pre-computed surrounding bbox (#253)

* First pass at pre-computed surrounding bbox

* Add some tests

* Remove console.log

* Typecasting is bads

* Add geography caching test

* Remove unused import

* Better destructuringz

Co-authored-by: Mark Maxham <max@ellis-and-associates.com>

* Update prettier config to enforce single quotes only in js/ts files, since we may want to have double quotes in yaml files. (#256)

* Update prettier config to enforce single quotes only in js/ts files, since we may want to have double quotes in yaml files.

* Making file formatting changes restricting to js and ts files

* [mds-config][mds-config-service] Remove mds-config and mds-config-service  (#254)

* Removing mds-config and mds-config-service, as these will not be part of the OMF contribution.

* Removing mds-config and mds-config-service as part of prepping for the OMF contribution.

* Reverting single quotes in yaml files.

* Reverting more single quotes.

Co-authored-by: Mark Maxham <max@ellis-and-associates.com>

* Allow services to specify TypeORM CLI options. (#259)

Use typeorm executable in ./node_modules/.bin

* Serializable service errors (#261)

* Use service provider interface for unit tests

* Move metrics service

* Move jurisdiction service

* Serializable service error objects

* Abstract processing of repository errors

* RepositoryError.GetProperty helper

* Use static error factory and type guards

* Minor refactor of type guard syntax

* Resolution of type errors that result from…
@schnuerle
Copy link
Member

Completed

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Catch-up from lacuna-tech/mds-core lacuna-tech/mds-core
2 participants
@jfh01 @schnuerle