Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orders Gateway #1396

Merged
merged 85 commits into from Mar 15, 2019
Merged
Changes from 74 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
707d80f
Electronic Orders: Update documentation strings to remove references …
jamesatheyDDS Nov 29, 2018
449f7d8
separate warrant officer ranks from lower commissioned officer ranks …
jamesatheyDDS Nov 29, 2018
95b1e15
Electronic Orders: add new needed columns and tables to the db schema
jamesatheyDDS Nov 29, 2018
20cc339
fizz doesn't like comments
jamesatheyDDS Nov 29, 2018
654826e
Electronic Orders: the orders table already has a status column
jamesatheyDDS Nov 29, 2018
e28a806
Electronic Orders: give losing_unit_id and gaining_unit_id differentl…
jamesatheyDDS Nov 29, 2018
a56dfc4
Electronic Orders: allow most of the new columns to be null strings
jamesatheyDDS Nov 29, 2018
9cca972
Electronic Orders: allow a few more columns to be null
jamesatheyDDS Nov 30, 2018
7dba7aa
Merge branch 'client-certs' into electronic-orders
jamesatheyDDS Jan 7, 2019
4b3927f
Restore simpler W-2 to W-5 abbreviations instead of CW-2 to CW-5, whi…
jamesatheyDDS Jan 8, 2019
15de51e
Add electronic orders fields to models
jamesatheyDDS Jan 8, 2019
837ab33
Merge branch 'client-certs' into electronic-orders
jamesatheyDDS Jan 8, 2019
4a5a8bb
Merge branch 'master' into electronic-orders
jamesatheyDDS Jan 9, 2019
1a2e606
Merge branch 'master' into electronic-orders
jamesatheyDDS Jan 10, 2019
76bed11
follow naming convention of other columns in the orders table
jamesatheyDDS Feb 6, 2019
851dddf
First draft of implementation of GetOrders endpoint. Many changes req…
jamesatheyDDS Feb 6, 2019
18570d9
Merge branch 'master' into electronic-orders
jamesatheyDDS Feb 7, 2019
fc340fe
store service member and unit info directly in order row, so that fut…
jamesatheyDDS Feb 13, 2019
7f73bc9
remove units table
jamesatheyDDS Feb 13, 2019
90fda4e
Put electronic orders into their own table for now; postponing the pa…
jamesatheyDDS Feb 14, 2019
584df9d
PostRevision and GetOrders working, but nullable / optional fields do…
jamesatheyDDS Feb 15, 2019
dc76b05
sort revisions in ascending sequence number order, to make appending …
jamesatheyDDS Feb 15, 2019
8e1aafa
Merge branch 'master' into electronic-orders
jamesatheyDDS Feb 15, 2019
93e9c46
reduce diff with master
jamesatheyDDS Feb 15, 2019
c6de256
remove Orders API <=> Internal API converters now that they are unused
jamesatheyDDS Feb 15, 2019
d854a68
implement PostRevisionToOrders
jamesatheyDDS Feb 15, 2019
29fc05c
check client cert for permission to read or write electronic orders b…
jamesatheyDDS Feb 15, 2019
038b747
generated swagger code checks the enumerations for correctness
jamesatheyDDS Feb 15, 2019
d513df1
deploy electronic_orders branch to experimental for a data transfer test
jamesatheyDDS Feb 15, 2019
8bc7f5c
update spelling of branch
jamesatheyDDS Feb 15, 2019
5ec7be8
The RNLT date is also optional, therefore nullable in the DB
jamesatheyDDS Feb 25, 2019
23a50ca
Merge branch 'master' into electronic-orders
jamesatheyDDS Feb 25, 2019
d1e4384
Merge branch 'master' into electronic-orders
jamesatheyDDS Feb 28, 2019
9dbde13
orders API v1.0 with new GET endpoints for orders by EDIPI and orders…
jamesatheyDDS Feb 28, 2019
2f1e8b0
Merge branch 'master' into electronic-orders
jamesatheyDDS Feb 28, 2019
765b1bd
fix build error, and don't deploy to experimental
jamesatheyDDS Feb 28, 2019
f66da3b
orders.yaml - update some doc strings
jamesatheyDDS Mar 4, 2019
2b50e7e
added tests for the ElectronicOrder and ElectronicOrdersRevision models
jamesatheyDDS Mar 7, 2019
17fe45e
added tests for GetOrders and GetOrdersByIssuerAndOrdersNum endpoints
jamesatheyDDS Mar 7, 2019
bd120f4
get EDIPI for SSN from DMDC
jamesatheyDDS Mar 7, 2019
48e6ea5
added tests for IndexOrdersForMember endpoint
jamesatheyDDS Mar 7, 2019
5577e3a
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 7, 2019
567de70
Use sub-tests to reduce overhead of DB truncate calls every test, and…
jamesatheyDDS Mar 8, 2019
338526f
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 8, 2019
75a2b37
make seqNum not nullable, since it's a required field, and that will …
jamesatheyDDS Mar 8, 2019
b490ef2
fix TestFetchElectronicOrderByIssuerAndOrdersNum (wrong order of para…
jamesatheyDDS Mar 8, 2019
1d1c1ff
checking in a blank line to make go-imports happy
jamesatheyDDS Mar 8, 2019
5c64a00
Make hasDependents not nullable - it's a required field
jamesatheyDDS Mar 8, 2019
56f1498
don't assume LosingUnit or GainingUnit structures will be present - t…
jamesatheyDDS Mar 8, 2019
0f7ffb3
PostRevision unit tests, both new and with an amendment
jamesatheyDDS Mar 8, 2019
0e940c4
add tests for PostRevisionToOrders
jamesatheyDDS Mar 8, 2019
4640aae
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 8, 2019
b16d854
Following "Querying the Database Safely" guidelines, make the FetchEl…
jamesatheyDDS Mar 8, 2019
9aef69f
Fix FetchElectronicOrdersByEdipi's slice-to-sliceOfPointers logic
jamesatheyDDS Mar 8, 2019
ca2349f
Log blocked attempts to read or write orders from specific issuers wi…
jamesatheyDDS Mar 8, 2019
417d773
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 8, 2019
f08e9c1
addressing PR feedback
jamesatheyDDS Mar 12, 2019
3b0d825
Revert adding x-nullable: false to the hasDependents field in Revisio…
jamesatheyDDS Mar 12, 2019
e012624
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 12, 2019
4578667
HasDependents is a bool* again in the ordersmessages.Revision, so upd…
jamesatheyDDS Mar 12, 2019
9a7bda1
Validate string enums using StringInclusion
jamesatheyDDS Mar 12, 2019
d574169
refactor client cert permissions checking for reading and writing ord…
jamesatheyDDS Mar 13, 2019
106ed0d
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 13, 2019
79042ec
Add URL to Appendix K-1 of the DTR to documentation of entitlements d…
jamesatheyDDS Mar 13, 2019
c85ffec
documents toElectronicOrdersRevision function
jamesatheyDDS Mar 13, 2019
6712dbb
Add indexes to electronic_orders table for orders_number+issuer and f…
jamesatheyDDS Mar 13, 2019
ea85cf7
small logging updates in the Orders API
jamesatheyDDS Mar 13, 2019
1fcd7ef
revert adding hs.DB().TruncateAll() to models.TestModelSuite()
jamesatheyDDS Mar 13, 2019
8b49cc4
Create an ElectronicOrder and its first ElectronicOrdersRevision in a…
jamesatheyDDS Mar 13, 2019
53b203b
Use master branch of golang.org/x/lint instead of travis-1.9 branch o…
jamesatheyDDS Mar 13, 2019
5f6e147
install the correct golint in 'make server_deps'
jamesatheyDDS Mar 13, 2019
44b1da4
deploy electronic-orders to experimental
jamesatheyDDS Mar 13, 2019
54fdbaa
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 13, 2019
03b5dfe
correctly format 10-digit edipi numbers into strings
jamesatheyDDS Mar 13, 2019
8713fcd
Verify orders read permission before hitting the DB in GetOrdersByIss…
jamesatheyDDS Mar 14, 2019
5262db5
use handlers.FmtDate{Time}Ptr() in payload creation function, like ot…
jamesatheyDDS Mar 14, 2019
6ba8986
Have the DB filter ElectronicOrders by allowed Issuers instead of doi…
jamesatheyDDS Mar 14, 2019
eaf087b
Remove migration that won't run on experimental. Will revisit creatin…
jamesatheyDDS Mar 14, 2019
c805896
reduce code duplication in creating ClientCert structs with all permi…
jamesatheyDDS Mar 14, 2019
666108a
clear the electronic_orders table so that adding indexes in a migrati…
jamesatheyDDS Mar 14, 2019
4df8b7b
Use handlers.ResponseForError() wherever possible in the Orders API i…
jamesatheyDDS Mar 15, 2019
a4d0318
Add new models.ErrWriteConflict error type, and use that instead of a…
jamesatheyDDS Mar 15, 2019
c4a7980
Add testdatagen.MakeDefaultElectronicOrder; refactor testdatagen.Make…
jamesatheyDDS Mar 15, 2019
dfb2957
Revert "deploy electronic-orders to experimental"
jamesatheyDDS Mar 15, 2019
651636b
Merge branch 'master' into electronic-orders
jamesatheyDDS Mar 15, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.
+3,324 −378
Diff settings

Always

Just for now

@@ -779,29 +779,29 @@ workflows:
- pre_deps_yarn
- acceptance_tests_local
# if testing on experimental, you can disable these tests by using the commented block below.
# filters:
# branches:
# ignore: placeholder_branch_name
filters:
branches:
ignore: electronic-orders

- integration_tests_office:
requires:
- pre_deps_golang
- pre_deps_yarn
- acceptance_tests_local
# if testing on experimental, you can disable these tests by using the commented block below.
# filters:
# branches:
# ignore: placeholder_branch_name
filters:
branches:
ignore: electronic-orders

- integration_tests_tsp:
requires:
- pre_deps_golang
- pre_deps_yarn
- acceptance_tests_local
# if testing on experimental, you can disable these tests by using the commented block below.
# filters:
# branches:
# ignore: placeholder_branch_name
filters:
branches:
ignore: electronic-orders

- client_test:
requires:
@@ -845,21 +845,21 @@ workflows:
- build_migrations
filters:
branches:
only: placeholder_branch_name
only: electronic-orders

- deploy_experimental_app:
requires:
- deploy_experimental_migrations
filters:
branches:
only: placeholder_branch_name
only: electronic-orders

- deploy_experimental_app_client_tls:
requires:
- deploy_experimental_migrations
filters:
branches:
only: placeholder_branch_name
only: electronic-orders
This conversation was marked as resolved by jamesatheyDDS

This comment has been minimized.

Copy link
@chrisgilmerproj

chrisgilmerproj Mar 11, 2019

Contributor

Before you merge let's remove these changes :)


- check_circle_against_staging_sha:
requires:

Some generated files are not rendered by default. Learn more.

Oops, something went wrong.
@@ -1,7 +1,7 @@
required = [
"github.com/securego/gosec/cmd/gosec",
"github.com/gobuffalo/pop/soda",
"github.com/golang/lint/golint",
"golang.org/x/lint/golint",
"github.com/go-swagger/go-swagger/cmd/swagger",
"github.com/codegangsta/gin",
"github.com/segmentio/chamber",
@@ -27,3 +27,7 @@ required = [
[[constraint]]
name = "github.com/99designs/aws-vault"
branch = "master"

[[constraint]]
branch = "master"
name = "golang.org/x/lint"
@@ -179,7 +179,7 @@ get_goimports: go_deps .get_goimports.stamp
server_deps: check_hosts go_deps build_chamber build_soda build_callgraph get_goimports .server_deps.stamp
.server_deps.stamp:
# Unfortunately, dep ensure blows away ./vendor every time so these builds always take a while
go install ./vendor/github.com/golang/lint/golint # golint needs to be accessible for the pre-commit task to run, so `install` it
go install ./vendor/golang.org/x/lint/golint # golint needs to be accessible for the pre-commit task to run, so `install` it
go build -i -ldflags "$(LDFLAGS)" -o bin/gosec ./vendor/github.com/securego/gosec/cmd/gosec
go build -i -ldflags "$(LDFLAGS)" -o bin/gin ./vendor/github.com/codegangsta/gin
go build -i -ldflags "$(LDFLAGS)" -o bin/swagger ./vendor/github.com/go-swagger/go-swagger/cmd/swagger
@@ -1071,7 +1071,7 @@ func main() {
ordersMux.Handle(pat.Get("/docs"), http.NotFoundHandler())
}
ordersMux.Handle(pat.New("/*"), ordersapi.NewOrdersAPIHandler(handlerContext))
site.Handle(pat.Get("/orders/v0/*"), ordersMux)
site.Handle(pat.New("/orders/v1/*"), ordersMux)

dpsMux := goji.SubMux()
dpsDetectionMiddleware := auth.HostnameDetectorMiddleware(zapLogger, v.GetString("http-dps-server-name"))
@@ -0,0 +1,20 @@
-- Commissioned officers and warrant officers are different paygrades.
-- They were conflated previously solely because they share entitlements,
-- at least for now.
-- Separating these ranks will protect us in the future from losing information
-- in electronic orders, will ensure that we address users respectfully with
-- the correct rank, and will make it simple to give them separate entitlements
-- if necessary.
-- At last check, none of the users who have enrolled in MilMove are warrant
-- officers. Therefore, it's safe and accurate to consider all users with
-- O_n_W_n ranks to be commissioned officers.
UPDATE service_members SET rank='O_1_ACADEMY_GRADUATE' WHERE rank = 'O_1_W_1_ACADEMY_GRADUATE';
UPDATE service_members SET rank='O_2' WHERE rank = 'O_2_W_2';
UPDATE service_members SET rank='O_3' WHERE rank = 'O_3_W_3';
UPDATE service_members SET rank='O_4' WHERE rank = 'O_4_W_4';
UPDATE service_members SET rank='O_5' WHERE rank = 'O_5_W_5';
-- Academy cadets are West Pointers, while midshipmen are Naval Academy cadets.
-- They were conflated previously solely because they share entitlements, at
-- least for now. That entitlement is not inherent to being a cadet, as
-- aviation cadets have a different entitlement.
UPDATE service_members SET rank='ACADEMY_CADET' WHERE rank = 'ACADEMY_CADET_MIDSHIPMAN';
@@ -0,0 +1,57 @@
create_table("electronic_orders") {
t.Column("id", "uuid", {"primary": true})
t.Column("orders_number", "string", {})
t.Column("edipi", "string", {})
t.Column("issuer", "string", {})
This conversation was marked as resolved by jamesatheyDDS

This comment has been minimized.

Copy link
@chrisgilmerproj

chrisgilmerproj Mar 11, 2019

Contributor

Can we add an index that connects issuer and orders number? If we do that query a lot I think it would help speed things up.

This comment has been minimized.

Copy link
@reggieriser

reggieriser Mar 12, 2019

Contributor

Along those lines, maybe make it a unique index so we don't somehow end up with duplicate orders_numbers from the same issuer.

This comment has been minimized.

Copy link
@jamesatheyDDS

jamesatheyDDS Mar 13, 2019

Author Contributor

I have added two indexes to the electronic_orders table - one unique that connects issuer and orders_number, and another not unique on edipi

This comment has been minimized.

Copy link
@jamesatheyDDS

jamesatheyDDS Mar 14, 2019

Author Contributor

@reggieriser Because of the bad data currently in experimental's DB, I have removed this migration (and the indexes it creates) for now. Due to a different comment, I redid how FetchElectronicOrdersByEdipi() works, so that index might need to be rethought. To address the issue on experimental, I can write a migration that truncates the table before creating the indexes. Since there's no important electronic orders data to preserve at the moment, does that sound like the best approach?

This comment has been minimized.

Copy link
@reggieriser

reggieriser Mar 14, 2019

Contributor

If there are no electronic orders (or revisions) to keep in any environment, then I think it's fine to delete those just before applying the indexes. Another option might be to write a query that would delete any duplicates that are causing the unique index to fail (maybe keep the newest electronic order only?) -- but that could be making assumptions that we don't want to make.

This comment has been minimized.

Copy link
@chrisgilmerproj

chrisgilmerproj Mar 14, 2019

Contributor

Let's clear them out, I think that's a lot cleaner as we deploy to all the environments.

This comment has been minimized.

Copy link
@jamesatheyDDS

jamesatheyDDS Mar 14, 2019

Author Contributor

There are two new migrations now - one that DELETEs from electronic_orders, relying on the CASCADE to delete all of the electronic_orders_revisions, and then a second to add two indexes - one on (issuer, edipi) and a unique one on (issuer, orders_number).

}

create_table("electronic_orders_revisions") {
t.Column("id", "uuid", {"primary": true})
t.Column("electronic_order_id", "uuid", {})
t.ForeignKey("electronic_order_id", {"electronic_orders": ["id"]}, {"on_delete": "cascade"})
t.Column("seq_num", "integer", {"default": 0})
t.Column("given_name", "string", {})
t.Column("middle_name", "string", {"null": true})
t.Column("family_name", "string", {})
t.Column("name_suffix", "string", {"null": true})
t.Column("affiliation", "string", {})
t.Column("paygrade", "string", {})
t.Column("title", "string", {"null": true})
t.Column("status", "string", {})
t.Column("date_issued", "timestamp", {})
t.Column("no_cost_move", "bool", {"default": false})
t.Column("tdy_en_route", "bool", {"default": false})
t.Column("tour_type", "string", {"default": "accompanied"})
t.Column("orders_type", "string", {})
t.Column("has_dependents", "bool", {})
t.Column("losing_uic", "string", {"null": true})
t.Column("losing_unit_name", "string", {"null": true})
t.Column("losing_unit_city", "string", {"null": true})
t.Column("losing_unit_locality", "string", {"null": true})
t.Column("losing_unit_country", "string", {"null": true})
t.Column("losing_unit_postal_code", "string", {"null": true})
t.Column("gaining_uic", "string", {"null": true})
t.Column("gaining_unit_name", "string", {"null": true})
t.Column("gaining_unit_city", "string", {"null": true})
t.Column("gaining_unit_locality", "string", {"null": true})
t.Column("gaining_unit_country", "string", {"null": true})
t.Column("gaining_unit_postal_code", "string", {"null": true})
t.Column("report_no_earlier_than", "timestamp", {"null": true})
t.Column("report_no_later_than", "timestamp", {})
t.Column("hhg_tac", "string", {"null": true})
t.Column("hhg_sdn", "string", {"null": true})
t.Column("hhg_loa", "string", {"null": true})
t.Column("nts_tac", "string", {"null": true})
t.Column("nts_sdn", "string", {"null": true})
t.Column("nts_loa", "string", {"null": true})
t.Column("pov_shipment_tac", "string", {"null": true})
t.Column("pov_shipment_sdn", "string", {"null": true})
t.Column("pov_shipment_loa", "string", {"null": true})
t.Column("pov_storage_tac", "string", {"null": true})
t.Column("pov_storage_sdn", "string", {"null": true})
t.Column("pov_storage_loa", "string", {"null": true})
t.Column("ub_tac", "string", {"null": true})
t.Column("ub_sdn", "string", {"null": true})
t.Column("ub_loa", "string", {"null": true})
t.Column("comments", "text", {"null": true})
}
@@ -0,0 +1,10 @@
add_column("client_certs", "allow_air_force_orders_read", "bool", {"default": false})
add_column("client_certs", "allow_air_force_orders_write", "bool", {"default": false})
add_column("client_certs", "allow_army_orders_read", "bool", {"default": false})
add_column("client_certs", "allow_army_orders_write", "bool", {"default": false})
add_column("client_certs", "allow_coast_guard_orders_read", "bool", {"default": false})
add_column("client_certs", "allow_coast_guard_orders_write", "bool", {"default": false})
add_column("client_certs", "allow_marine_corps_orders_read", "bool", {"default": false})
add_column("client_certs", "allow_marine_corps_orders_write", "bool", {"default": false})
add_column("client_certs", "allow_navy_orders_read", "bool", {"default": false})
add_column("client_certs", "allow_navy_orders_write", "bool", {"default": false})
@@ -0,0 +1,2 @@
UPDATE client_certs SET allow_army_orders_read = true, allow_army_orders_write = true WHERE subject SIMILAR TO '%[Aa]rmy%';
UPDATE client_certs SET allow_marine_corps_orders_read = true, allow_marine_corps_orders_write = true WHERE subject SIMILAR TO '%(usmc|Marine Corps)%';
@@ -0,0 +1 @@
change_column("electronic_orders_revisions", "report_no_later_than", "timestamp", {"null": true})
@@ -0,0 +1,2 @@
add_index("electronic_orders", ["issuer", "orders_number"], {"unique": true, "name": "electronic_orders_index_by_issuer_and_orders_number"})
add_index("electronic_orders", "edipi", {"name": "electronic_orders_index_by_edipi"})
@@ -165,7 +165,7 @@ func (suite *BaseHandlerTestSuite) AuthenticateTspRequest(req *http.Request, use
return req.WithContext(ctx)
}

// AuthenticateDpsRequest authenticates TSP users
// AuthenticateDpsRequest authenticates DPS users
func (suite *BaseHandlerTestSuite) AuthenticateDpsRequest(req *http.Request, serviceMember models.ServiceMember, dpsUser models.DpsUser) *http.Request {
session := auth.Session{
ApplicationName: auth.MilApp,
Oops, something went wrong.
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.