Product Architecture Planning

[FilledStacks]

As mentioned at the end of the new video you can watch here this is the discussion where you can ask questions about the Technical Product Architecture. If you have suggestions please leave them below. The process of determining the best technical architecture is the same one we followed to determine the high level overview of the product architecture. We will walk through the largest and most important use case and technically detail what needs to happen or what would be the best way to do this. The scenario is as follows:

1. User adds product to their cart
2. User wants to checkout
3. User has to add a payment method
4. User checks out
5. Order shows up on restaurant app
6. Restaurant confirms the order and sends a preparation time
7. Restaurant marks order ready for pickup
8. Driver gets the order
9. Driver delivers to the User

These are the high-level steps for the most important process in this service. We will use this to determine the following technical details that would be required before we start writing any code.

• Establish how the data will flow from a user client, through the services and back to the client
• Establish which data models will be required to get the information from the client through the services and back to the client in the most effective manner
• Establish the data collections that will be required to provide this information
• Establish the usage of the cloud functions and their intentions

Once we have all this information we will have a better idea of what we are building which will give us a better platform to work off when planning the actual development work. Once the decisions have been made all of this will be moved into a document in the Wiki which will detail everything that has been decided.

Please share your ideas below if you have any.

[FilledStacks]

I'm going to paste some of the private discussions and comms I get here below so that I can use it while doing the new planning. I like using the word resources instead of groups! See below.

This is from Fernando who is on the FilledStacks team.

TLDR;

• Order Management: /order/[endpointName] ==> /orders
• User Management: /user/[endpointName] ==> /users

---

I saw the last episode of boxtout, excellent! Is a great project, I will try to help when I have some space on my free personal time. Only some suggestions that you could take or not. In the section How will we manage functions as a service of the technical planning and overview, for what you call as groups, I think will be better to take the REST api conventions and call them resources. And, to continue with this convention, the name of the resources on the endpoint should be plural noun on all of them, to have some consistency.

Order resource example:

• GET /orders - Retrieves a list of orders
• GET /orders/12 - Retrieves a specific order, in this case the order with ID 12
• POST /orders - Creates a new order
• PUT /orders/12 - Updates the order Streamline the process of setting up the backend #12
• PATCH /orders/12 - Updates partially the order Streamline the process of setting up the backend #12
• DELETE /orders/12 - Delete the order Streamline the process of setting up the backend #12

I hope this helps, it is to contribute with my grain of sand 😁

[FilledStacks]

This is from Rodrigo over on LinkedIn. Great suggestion around using Express to keep 1 endpoint facing publicly with the redirecting happening internally.

Hi Dane,

I've been following the Box't Out project you are developing and wanted to share some thoughts with you if that is ok.

I saw you wanted to use cloud functions to contain each manager (or service), which makes perfect sense. That made me think that you will probably want to call those managers from within another manager to fetch some data or do some calculation. Let's say you want to create a new user. You might hit the Users Manager service endpoint to achieve that; however, let's say you also need to create a Payments Manager customer. That would mean you need to make separate calls from the client-side or make another HTTP request from the User Manager cloud functions itself.

After thinking about that, Express came to mind. What if instead, you have one cloud function that contains all your microservices. Using express you would be able to expose all your endpoints like before, but, you would also be able to call different managers within a manager itself, which leaves the client with a very simple endpoint to hit whenever it needs something.

Using express also allows you to take advantage of a lot of middleware that can make our lives easier. For example, logging and even authentication.

I've attached a simple diagram better illustrating what I'm talking about. Let me know what you think!

[rodrigonso]

[rodrigonso]

I did some more thinking and came up with a good hybrid approach. Creating one cloud function for each endpoint that you want to expose, and inside each of those functions, use Express to route requests regarding that endpoint.

For example:

GET /usersApi/getAllUsers

/users is a cloud function
/getAllUsers is a route handled by express inside that cloud function

we could replicate that to any other endpoints we want to expose.

[FilledStacks]

@rodrigonso yes, that was going to be the way we do it if we needed the endpoints. At the moment there's only about 3 api endpoints that I know we'll need. Checkout (Orders), addPaymentMethod (Payments), confirmPayment (Payments). The rest will be reactive functions that react to certain information being added, removed or updated in firestore. More will come, so we could possible write a snippet to create one of the express server endpoints.

Will look into the benefit of having that over the other ways when we get to the implementation.

[FilledStacks]

We might have to scrap the express functionality. I'm struggling to write the code in separate files per endpoint and have it set on the same express server. I've written this to automatically collect all .function.ts files and add it to the exports to deploy.

This means we can write our http.request functions the same way and deploy automatically. I don't have enough experience to do that using express.

@rodrigonso you have any idea how we can do that? Basically I want the folder for each resource's idle function to contain 1 file per api endpoint. that way it'll be easy to maintain in the long run as we add more endpoints.

With that comes the idea of only having one express app and attaching those endpoints to it. I don't know how to do that using type script so the best thing would be to use firebase http functions directly with one function per endpoint, if I can't find a way with my alotted 2 hours tomorrow to figure it out.

Let me know if you have any insights into that.

[rodrigonso]

@FilledStacks I haven't taken a look at the code yet, but I will tomorrow when I have more time. But from my understanding, you are exporting the cloud functions on-demand based on a naming convention, right?

If that's the case, we need somehow to have a requestHandler(req, res) function on every resource that deals with the cloud function request and re-routes said request to the function that is supposed to fulfill that request. We also need to pass a reference to the express app to each resource.

[FilledStacks]

Maintaining carts: Remote

Lets tackle number 1: User adds product to their cart

The first and most straight forward idea that I can think of when it comes to the users cart will work as follows. When a user is created we generate a cartId that is attached to their profile. At the same time we create a document in the carts collection which will belong to the user. When we add to cart it goes into this cart, when we checkout this cart gets cleared and all the info is dumped into the order information. The user basically has one cart throughout their lifetime on the app that gets re-used and we know exactly what the cartId is at all times.

This makes it so that adding to cart is quite simple in terms of a logic stand point. We can add directly from the app into the firestore DB for the users cart. We can either add is as subcollection or we can add it as an array of items on the document itself. To keep things simple I think the best thing to do would be to add it on the document as an array. This will allow you to always get all the information back in one single call.

Total cart cost

The obvious way to keep the total cost in sync is to update the price, from the current, based on the action being performed by the user. This includes tax (which will be added to the users profile, depending on how their regions tax works). When a user adds to cart we will make the update call to add the product into the product list on the cart as well as update the price to add the new total to the current total. This functionality exists in the Firestore package to perform atomic updates to values on firestore so we'll use that. So we'll do an update for the total which we will then read from the DB again and set as the new cart locally. This should keep the total and the entire cart in sync locally. I hope

Maintaining carts: Local

I wanted to play around with the idea of a local cart on the device. This will reduce the number of updates in the firestore DB and also will allow us to keep the cart local until checkout meaning we don't have to track it on the backend. I've never done it this way and I prefer tracking everything on the backend. Doing it this way means that if we build a web client they will have to replicate the cart functionality for the application locally as well.

So we'll use something like hive to build the cart up locally, this will allow us to keep track of the products in the cart as well as the totals for it. the good thing about this is that there's less writes and reads from the DB. This appears upfront to be easier than keeping it on the backend and syncing it up. So I'm leaning towards this option. Lets see where the rest of the steps take us.

Please leave replies in the thread for this message on this discussion if you have any thoughts on any of this.

[ketanchoyal]

Maintaining carts locally means that the items in the cart will not be persistent across multiple devices if the user has logged in to multiple devices using the same account.

If the cart is maintained locally:
Doubt: What if the user tries to order from multiple devices at the same time? (this is the worst case possible) then we have to think about it as well

Possible solution:

1. We can restrict multiple logins
2. We can maintain the carts remotely

PS: I'm not an experienced developer like you guys, I just thought of this case so I asked

[FilledStacks]

@ketanchoyal good observation! I kind of forgot about the local only downfall. That is exactly the scenario / problem that kind of rules out the option. We'll keep the cart on the backend to cover the cart sync scenario.

Thanks for adding to the discussion, I appreciate it.

[adamschnaare]

One thought that may be a blend of the two might be:

When Cart is updated in the client app

1. Update the local store of data representing the cart
2. Send the update request to the backend (to maintain record of the transaction)
3. Only pull new data from the backend when navigating or reloading the view

This may keep the calls over the network lower, ensure a fast UX, and keep things in sync across devices / experiences.

[FilledStacks]

Adding a Payment Method

This is point number 3. I realise that number 2 isn't actually a step in code but more of a feeling haha. Before a user can checkout they need to have a payment method that the money can be deducted from. We'll need to find a payment provide when the time comes to make use of. PayPal v2 api and stripe is not available in South Africa so I'll probably as someone in the team or that I work with to create a stripe account for me that we can use. So that will be left to later. The one thing that we can tackle is how the payment method will be stored and where that will be.

When adding a payment method from the app we'll add that into the payment processor we'll be using and request the token associated with that payment. We'll put that payment token onto the users account as preffered payment method as well as into a collection of payment methods on the users account. This will be so that we can keep track of multiple payment methods for the user.

[adamschnaare]

Will each restaurant be able to determine what currency(ies) they accept? Will the payment processor allow for that flexibility?

[FilledStacks]

@adamschnaare yes. This will most likely be determined by the location that the user has selected and the territories that we've assigned. There won't be a different currency per user, but rather a currency per country. So if the user orders in that country we'll use the dominant currency in that country.

[FilledStacks]

Checkout process

Okay here we go, this is quite a big one. The checkout process, this covers step 4 in the steps laid out in the discussion original post. Lets start out with the way that we want to use this in the code. I want to call a single api call on the backend for /order/checkout and post the cart json. Inside this cloud function we will then:

1. Use the payment method token attached and request the payment from the payments resource. This will be done using a basic http request.
   1. On the payment processor: We need an api endpoint that will return to use the payment receipt given the response from the third party processor. if the receipt is not returned it has to return an error message detailing what the problem is. This message should be formatted in such a way that we can return it to the user to show in the application.
2. If the payment failed we show the error returned to the user and bail on this process and return the error message in a consistent format. See response format below
3. if the payment passed we create the order in the orders collection that contains the entire cart with all its information and the starting status.
4. When the order is created we have two options:
   1. Send the push notification from this function to the restaurant app. This means we'll have to mix all the code into this function which I don't like.
   2. The second option is to create a cloud function that's attached to the orders collection that will listen for new orders being created and then sends out a push notification using FCM. I like this option more because it keeps the push notifications responsibility separate from the checkout functionality. This will also allow for better scaling because the original call can return when ready instead of awaiting the sending of the push notification as well.
5. Return the order ID in the data when successful.
6. On the client: Clear the navigation stack and navigate to the OrderStatusView passing the id returned from the checkout request.
7. On the client: In the OrderStatusView listen to the real-time updates for the document matching the order id orders/[orderID] . This view will always perform this action to allow for status updates to be seen as they happen.
8. On the restaurant app: We are subscribed to orders that match the restaurants id's in real time. Those will be updated in real-time as the new orders are added.

Using the steps above we can populate an cart from the users device onto the restaurant app. This is a big step to get over. This allows us to then also track on the device the orders status as well as receive updates for it. See below the proposed response format for api resposes in our resources.

Response Format

We should have a consistent response format that all api requests can serialise to. This will make it so that it's easy to determine if there's an error and get the error information as well.

No error

{
	"data": {
		"orderId": 12345
	},
	"error": null,
}

With error

{
	"data": null,
	"error": {
		"message": "The payment method used does not have enough funds"
	}
}

[francescopasin]

Regarding the response object, you can use specific http status codes for errors, so you can catch then in the client. For this reason it couldn't be necessary to return error: null or data: null since the two cases would be differentiated by status codes.

[FilledStacks]

@francescopasin Those codes will be in the http response itself. The data above shows what'll be in the content of that response. I prefer managing my messages from the backend to ensure less client side logic. We'll know more accurately on the backend what caused the problem so we can send back a friendly error message to be displayed as is in the client app. This would avoid conditionals and allow us to do error handling for api problems in one place instead of at every request.

[ketanchoyal]

@FilledStacks One more thing, if we add error messages straight from the backend then there is going to be a lot of trouble if we want to add Internationalization into our app in the future.

[FilledStacks]

Restaurant App

Steps 5-9 all happens from the restaurant app and driver app. The customer mobile app will already be reachable through direct notifications using the latest device token as updated on the start of every launch. In addition to that the client application will be listening to the order in real time that's being viewed. This leaves us only the restaurant app and services involved to detail for these 3 steps.

Order updates

When the order shows up in the restaurant list through the real-time query stream the user can tap on it and then mark it as preparing and send a time in minutes. When this is done there will be a cloud function listening to updates on orders (mentioned already in the above post) that will check the latest status, and send out the appropriate push notification. We might have to maintain two collections of orders, one with the full orders, and one with a simple orders list that can listened too in the listing views so that we don't read all that information. But we'll look at that later on

Once the status is updated to Preparing and we have a time we let the user know their status has been updated. When the order is ready we go through the same process as above. The restaurant app updates the order on firestore, the cloud function sends out the push notification and gets the drivers involved.

Once the order reaches a certain status (ready for pickup) we send it out to the closest drivers (at the beginning, just all the drivers in the area) and the one who accepts the order first will be assigned to it. They then get the order with the delivery address and pick up address and do the rest.

The driver app will also be allowed to update the status of the order. They need to mark it as picked up as well as delivered and be able to accept a tip (which we'll plan later). And that's basically it. Next up we'll go over the technical insights discovered through this process.

[Diegi97]

I think you should send the order to the drivers a few minutes before it's ready for pickup, maybe 5 minutes if the closest driver is 2km away from the restaurant, 10min if the closest driver is 5km away or something like that. To keep the order the least time possible on "ready for pickup".
Keep the great work!

[FilledStacks]

@Diegi97 good suggestion! I'll keep that in mind. I appreciate the input.

[jorgemndoza]

We also need a way for driver to set their availability. Something like an on/off switch to toggle if they are available to work.
I don’t think we want to send notifications to drivers when they are not working.

[FilledStacks]

@jorgemndoza You are correct. We'll go over those implementation details when we get to the implementation.

[FilledStacks]

Technical Insights

So what did we learn from doing this? Well the first thing is that we'll have different types of cloud functions which we can categorise and locate accordingly in the source code. We have:

• Reactive functions: These functions are listening for changes and will only run whenever any of those changes occur. The common ones are listening to a new entry in a collection or listening to an update in a document. This will be used throughout the backend to keep things separate and easy to extend.
• Idle functions: These are functions that will stay idle unless called explicitly from an http request or through the cloud functions sdks that are available. This is seen in the checkout request and most likely through the function that will help us add a payment method into our system for a user.

In addition to that we have uncovered a few of the concepts that we'll need to details out now to get this architecture to work properly. We'll go from the top down and see what we'll require in terms of the modelling of the data to be able to achieve what we want.

Data models and collection structure

Lets start with the most basic things. Everything on the restaurant side.

• Restaurant (Producer): This model represents the entity that's providing the product the user is ordering
  • Name
  • Address
  • Opening Times
  • Menu's
• Menu: This is the object that will represent a list of products and their options available from the Producers
  • Availability
  • Products
• Products: The products represent actual things that the user can order and pay for through the app.
  • Name
  • Price
  • Description
  • Availability
  • Options
• ProductOptions: The options that can be added to modify a Product in the cart. Additional cheese, a flavor, a drink with the meak, etc.

Those are the models that we'll require from the restaurant side. Next up lest look at the models that will be required from the user side.

• User (Consumer): This represents the entity that will be ordering the product and paying for it.
  • Name
  • Address
  • Payment Method
  • Account details

That's the only major model I can identify at this point. Of course the models used to represent things like the Address, PaymentMethod and AccountDetails will be models as well but those aren't important in carrying over the data from the client to the restaurant and back. Those can be stored as strings if we're lazy (which we are not). But that's how I determine if they're worth mentioning in this document. Next up is the meat of the process. All the models involved in connecting the consumer to the producer.

• Cart: This represents the current set of products that the user would like to buy from the producer.
  • Products: A list of all the products with their options that the user would like to buy
  • Total: the current total of the cart
  • UserId: the user the cart belongs too
  • RestaurantId: The restaurant the orders are from
  • DeliveryLocation: The place to where the user is ordering the food
  • Delivery instructions: Any additional instructions for delivery that will be given to the driver.
• Order: This is the "promise" made by the system between the Producer and the Consumer that describes what was ordered, what its state is, and who it belongs too:
  • Id: The is of the order which will match the document id.
  • status: The status of the order to tell the user and the restaurant what state the order is in.
  • Delivery address: The address where this order must be delivered too.
  • Pick up address: The place where this address must be picked up.
  • Products: All the products that were ordered along with their options
• Delivery: This is the model that will be used to track the driver delivery contract. The same way an order is the contract between the user and the producer. The Delivery is the contract between the Producer, the User and the Driver that says "I'm taking this order to the user, and this is my current status". This will be updated in real time with the lat long of the driver
  • OrderId
  • DriverID
  • UserId
  • RestaurantId
  • Lattitude
  • Longitude
  • Delivery Address
  • Pickup Address
  • Time picked up
  • Time delivered

That's all the major models required that we can identify based on the most important process in the entire product. Getting what the user ordered to them in a manner that keeps every party involved up to date with what's happening. These are not "The complete models" we will be adding and taking away properties as we develop but this does give us an idea of what we have and will need going forward. This now makes it so that we have 2 of the initial requirements for this document complete.

• Establish how the data will flow from a user client, through the services and back to the client
• Establish which data models will be required to get the information from the client through the services and back to the client in the most effective manner

Next up we have to go over the collections that will be required on firestore to allow us to move data how we described above. That's up next.

[francescopasin]

Why are "delivery address" and "instructions" in "cart" model? Usually this informations are given during checkout (usually saved in user account) so they could be only inside "order" model. Do we need it inside "cart"?
Also since "delivery" model is associated with an order isn't the delivery address there a repetition?

[FilledStacks]

@francescopasin Yes, we do need it on the cart. The cart will get used to create the order. So when we checkout we submit the entire cart (given that each cart can be delivered to a different location). At that point we take the delivery information and add it into the order and destroy the cart. That will then be on the order.

The delivery address will only exist in one place at a time, well two I guess given that we'll have multiple on the user model as well. This will allow us to present the user with a list of addresses they've added where we can deliver to.

[Anticonformiste]

I guess the delivery address[es] should be saved on the Customer model. On the User.Adress we'll get their main address, and on each order checkout (if he wanted an address != than his main one) we will add it on his model (ex => secondaryAddresses: List[]).
Then the chosen address will be passed to the Delivery model on the field DeliveryAddress.
This way we'll avoid repetitions and our data flow stay fluent and consistent (The customer next time, will just choose among his frequent addresses.

[FilledStacks]

@Anticonformiste yes, that's what I was thinking by adding a collection of addresses under each user document. In the document itself we'll simply keep the id of the address that's default, and locally in the client we'll send the selected address with the order placement. In addition to that on every start the address will start up with the default address (as indicated on the user profile).

[FilledStacks]

Firestore Collections

In this section we go over the collections we'll require to get the information from the client through the system and back to the client. We can follow the same path as what we did with the models above. We'll look at the collections the restaurants will need.

• Restaurants: This will be a collection of documents that contains the description and details of restaurants.
  • Menus: This will be a sub collection in a restaurant document that contains the menu's for that restaurant
    • Products: This will be a sub collection in a menu document that contains all the products for that specific menu.
      • Options: This will be a sub collection in Products that contains all the options available for that product.

The structure above will allow use to apply well defined security rules from the highest level down and also allow us to easily find products / options for specific restaurants with their specific menu's. Next up is the user:

• Users: A ****collection that will store the information of the user as a document.
  • PaymentMethods: A collection that stores all the payment methods that the user has added to their profile
  • Addresses: A collection that stores all the addresses the user has added to their profile as documents

The we can move onto the collection required for these two to communicate with each other and make their exchange.

• Carts: A collection that houses all the carts as documents
• Orders: A collection that keeps all the orders that has been placed through the system
• Deliveries: A collection that keeps the delivery document for the real-time system to track it.

That will cover all the collections for this process. With these collections we have sufficient points of interception in case we want to add additional functionality / functions to do more. It covers our scenario and if we can get away with only these collections we'll have a very lean system on our hands to maintain. Which is great for the long run. With this we have covered

• Establish the data collections that will be required to provide this information

And I made a mistake above. We actually covered the last point which is Establish the usage of the cloud functions and their intentions in the part above this. The first portion under Technical Insights. Which concludes my rough planning.

I hope this is insightful. I will use all the posts form this discussion and create a well formatted document that has everything in it which we will discuss in the next video coming out next week.

[FilledStacks]

After the research I've found that multiple mobile apps connected to one firebase project is possible. Drawbacks of that is

• Facebook login doesn't work since it provides access app wide
• Some known issue with dynamic url's

Other than that it looks promising. We'll connect all our apps to one project, each project will update their user object with what it belongs too after its created. This means that for all the users we'll have one model and then use a type value to distinguish between them.

If this becomes too cumbersome then we'll instead of updating the user object with a type, as soon as the new user profile is created we'll trigger a function from the app being used that will take that user details and move it to a new collection dedicated for those users. These collections will be:

• Drivers
• Producers

Given how much work the second suggestion is we will probably do it from the beginning. I'm going to decide on that today before we move on to the second part.

[vancegillies]

This looks awesome, I would maybe consider de-structuring the data for the menus?
what if in the future and restaurant what's to have a day time menu and a night time menu be able to restrict the menu options on the fly? I'm imaging having and array of 'menus' on the restaurant object that contain an array of products

• Restaurant
  - Menus
  - product ids
  - Products

This would allow future features like

• Multi menu (seasonal/time of day/staffing cap)
• Special items that are in the system but not in any menu thus not available.

[FilledStacks]

@vancegillies that's a good idea. We can definitely do something like that. Most of this will develop as we go through the building of the functionality. The menu's will be a collection on the restaurant. Each of it will have its own availability. When requesting a menu we'll ask for the current one that's open at the local time for the user. So a menu will have multiple menu's. breakfast, lunch, dinner, Weekend special, Wednesday burgers only etc.

[johnstef99]

Why are we choosing subcollection for the product's options and not an Array of Map<string,dynamic>. We could get one less read that way. Is it because we will load all the product's options that we are not going to use? Is that so time consuming ?

I'm not always sure whether to go with Array or subcollection..

[FilledStacks]

@johnstef99 we could go either way. I like using the array of maps too. I have to read up a bit more on what firestore recommends and how we can set it up properly. I also think putting the options in an array of map is the best approach. Especially since they won't be listened too in real time. It'll be a once off get request and most likely won't change so the caching will take over after that for at least 28 days.

Good call.

[taufeeq849]

Hi, Dane and the Filled Stacks Community. Firstly, I think this is a fantastic initiative.

I'd like to discuss the reasoning behind choosing the Firebase suite of products as well as the purpose of cloud functions.

To begin, in your video you mentioned that Firebase was chosen due to this being a small team project. Firebase certainly offers an 'out of the box' infrastructure requiring very little to no setup cost. However, surely this comes with the downside of less configurability and lower performance when compared with the ''traditional' way of doing things: a REST API with an underlying SQL database? To give you an example, say that an end-user is attempting to search for a restaurant based on their name. Ideally, the logic for this should be server-side. Thus, using SQL this would be a simple 'like' query. Whereas in Firebase, as I understand it, the logic for this function needs to be client-side or you have to look at a third-party solution such as Algolia. For reference, refer to these docs: https://firebase.google.com/docs/firestore/solutions/search.

Consequently, how you do balance the trade-off between the lack of functionality of Firebase versus going with an infrastructure that requires longer to set up?

On a different note, what purpose will the cloud functions serve? Will the cloud functions handle all database querying so as to abstract the underlying Firestore storage? Or will the cloud functions only handle queries that the Firestore engine is not capable of: such as the search query mentioned above. The reason for this question is because in a 'traditional architecture set up' with a server and SQL database - the client interface would not access the SQL database directly. Rather, the database would be abstracted through an API. However, Firebase offers Firestore security rules, which enables the client to access the Firestore database securely.
Therefore, will the client query the database through the use of cloud functions or will there be a balance between the two?

[joekendal]

I didn't know even-driven is a thing right now,

Since you mostly work on the client-side, a lot has been happening in the back-end that it's become so complex to actually work on when following the next-best things each year. AsyncAPI is hopefully the thing that makes it easier going forward

[FilledStacks]

@joekendal I didn't know you were trying to convince me haha. Why would you tell me to use something I have no idea how to use, it would make this project end horribly. I only have 1 and a half days every week to build this. There would be no time to learn new patterns and systems during that. It would probably make it a horrible experience for me. Which luckily I don't have to go through.

The goal of this project is to teach developers how to build a large system. These same things I'll be doing in this project can be applied to any toolset for any project. Even doing a costly migration would be a good thing down the line. If my goal really is to help a million developers become senior software engineers then they'll better be able to recognize when they need to jump ship and then also how to do it. Great opportunity for me to carry over more knowledge.

I'll get some AWS backend engineers involved, most of them actually also don't want me to use Firebase haha, but the tools don't really matter right now. They can help me make better decisions with the tools at hand.

[FilledStacks]

@joekendal AsyncAPI looks dooooooope!

[taufeeq849]

@FilledStacks Dane, thank you for the fantastic insight. Yes, I think firebase is the right path to go down especially considering the time allocated for development and how quickly you are able to build with it. I appreciate how you have walked us through your thought process when it comes to choosing the tools and services required to build applications. Looking forward to this!

[taufeeq849]

@taufeeq849 I think you should look into GraphQL instead. RESTful APIs to an SQL query don't make sense these days. You're better off using something like AppSync to make fast serverless queries from the client. Don't bother running your own SQL database & REST api. I hope this helps.

@joekendal Thank you for the response. I have used GraphQL briefly and the developer experience was much better than having to set up and maintain a REST Api with a SQL database. However, there is a lot of GraphQL service providers. Namely, I have used postgraphile with a PostgreSQL database. But I am aware of services like Prisma and Hasura. In your experience, which GraphQl providers would you say is best?

[rodrigonso]

Real-time delivery updates

I think for a service such as this, having the ability to track the progress of an order/delivery is fundamental.

Customer-side

The customer wants to track their food as soon as that order becomes "active". We can define active as whatever we want, but a good start is when the order has been confirmed by the restaurant. I believe you guys already mentioned push notifications as a means to notify customers of delivery milestones (i.e. picked up food at a restaurant); however, enabling the customer to open up their app and see in real-time where their food is, complemented by push notifications, would bring a lot of value to the user.

Restaurant-side

I think the same concept applies here. It is important for the restaurant to know where the delivery driver is when an order has already been assigned to that delivery driver. This would give the restaurants more transparency on the delivery cycle and enable them to contact the customer in case something goes wrong for example.

How to implement this?

Real-time data is fairly easily achieved with Firebase, for that reason, I don't think it would be too much of a pain to implement this. However, one thing we need to take into consideration is how we are going to listen to the delivery driver's location. There are multiple ways we could achieve that, but we should keep battery life, data costs, read/writes frequency in mind.

Approach 1: Listen to device location at all times (expensive, but very accurate position data)

Approach 2: Listen to significant device location changes (cheap, but not very accurate)

Approach 3: ?

Geofencing

Another feature I thought this service could benefit from is Geofencing. Instead of depending on the delivery driver to press "Here" or "Picked up food" buttons to update delivery status, we could use geofence events to automatically trigger whatever functionality we want (including sending push notifications to the end-user), and also prevent drivers from saying they've arrived when in fact they haven't for example.

How?

We can use the device's geofence API to setup geofence regions along the world and listen to entry/exit events of those regions. This is a native feature, but there are many packages available that can be used to achieve this.

Example:

1. User creates an order
2. Restaurant confirms the order
3. Delivery driver is assigned
   • at this point, we set up a geofence region around the restaurant and listen to entry/exit events
   • we only allow the driver to press "Here" or "Picked up food" if the driver is inside that geofence region
4. Delivery driver enters the restaurant georegion
   • this triggers an entry event which then notifies the restaurant that the driver is almost there.
   • the driver can now press the "Here" or "Picked up food" button.
5. Delivery driver picks up the food and start's driving to the customer's location
   • at this point, we setup a geofence region around the customer's address and listen to entry/exit events
6. Delivery driver enters the customer's georegion
   • this triggers an entry event that notifies the customer that the driver is approaching.
   • the driver can now press the "Here" or "Delivered" button
7. Food was delivered, we stop listening to events.

[FilledStacks]

@rodrigonso You're right on the geo-fencing thing. We could probably just use 1 for when the driver is 50m away from the house. All of the other updates will be "manually" triggered by our system. We'll send notification on every status change so the parties involved will basically tell us when to send it.

And in terms of the background, we will definitely do that, but from what I've seen the companies tell drivers to keep their app open if they want to get a higher rating. Which is definitely the case. We're going to do background updates but iOS and android recently will limit you after a while. So to avoid that they'll have to keep the app open.

We can add the extra work, or they can ensure they provide updates accordingly for their delivery to ensure the most real-time information we can get. At first we'll most likely require the app to be open and then update that functionality later on.

[rodrigonso]

Right exactly, use the geofence service only at milestones like arriving at the restaurant.

So for leaving the app open, do you mean the app is running in the foreground? I don't know if the locations package is able to run in the background (while the app is not closed) if that is not the case, the driver won't be able to get directions on a different app like google maps. If it can, then we are good!

[FilledStacks]

@rodrigonso they can still get the directions on a different app. It'll run while it's in the background. It should be possible when the app is closed as well. I've done that with native code before.

[adamschnaare]

How will the system know the Drivers' locations / proximity to new Orders from Restaurants?

• Will the Driver app react to all new Orders and then check for proximity?
• Will the Driver app actively update their location in Firebase so that new Orders can notify drivers that are close?

[FilledStacks]

@adamschnaare we'll update the drivers location every few hundred meters when they're not delivering. That way we can query at any time if they're close. We'll build a function on the backend that will get the nearest drivers to the restaurant and send them a notification.

[rodrigonso]

Delivery ETA

Due to the nature of this service means that we need to be very fast at delivering the food. However, that is not always possible. Maybe there is not a close driver around the restaurant in question, or the restaurant is understaffed at that particular time. These things happen and are inevitable. However, one thing we can do is to provide the customer with real-time updates as I mentioned above, and the most accurate delivery ETA possible. These features all serve the purpose to minimize customers' anxiety.

Calculate distance?

One strategy to give the customer an ETA is to calculate the distance between driver -> restaurant and restaurant -> customer, however, this is wouldn't be accurate for many reasons. One of them being the fact that we are not taking into consideration the time it takes for the food to be ready. In the best-case scenario, the time it takes for a driver to drive to the restaurant is the time it takes for the restaurant to prepare the food -- but that is not always the case.

Machine learning

A better approach to this problem, in my opinion, is to harness the power of machine learning to give us the best possible delivery ETA to our customers. To do this, we will some data. To make it simple, we can find some open-source data collections on this (I'm sure there are plenty out there) and use that to train a machine learning model.

Parameters

These are just my suggestions, please feel free add/suggest.

To better understand how we can create a decent prediction model, we first need to understand what determines the time it takes to prepare the food?

1. Day of the week (weekends tend to be busier for example)
2. Time of day (lunch time, etc)
3. Number of staff
4. Menu item (burgers are fast, steak dinner not so much)
5. Size of order (num of items)

These are just a few I could come up with. But gathering data on x amount of restaurants for these given parameters would allow us to create a pretty decent machine learning model, which in turn would enable us to provide our customers with the most accurate delivery ETA possible.

Accurate Delivery ETA

Finally the accurate delivery ETA would be composed of three variables:

DistanceA: Distance from driver to the restaurant
DistanceB: Distance from driver to customer
FoodPrep: Output from machine learning model

Let me know what you guys think!

[rodrigonso]

Those are very good points, y'all. I agree that instead of wasting development time building this feature now (with the very limited data we have), we should instead think of ways we can collect and store the data we will need in the future to implement this feature.

For instance, we could store the time it takes for the driver to press the "Picked Up Food" button after pressing "Arrived at Restaurant" button. We could also include other information about the order in this data such as order size or something around these lines.

If we really want to implement this in the future, It would be important to think about these things right now so we can collect data from the get-go, and have a nice pipeline setup.

What do you guys think?

[adamschnaare]

Late in the game again, sorry. But if I'm understanding it correctly, the Restaurant will have to accept the order and mark it ready for pickup right? Since we know the contents of the order, and the time between these two actions, that gives the resulting time of prep, correct? That would be what is used to train the ML program, correct?

[FilledStacks]

@adamschnaare yes, that data will be collected from the beginning so that we can easily calculate this info when we need it for training data. It's done by saving all the times the status's changed as a time stamp. That way you can go back through all orders and calculate the exact time the status's were changed and updated per week, even per hour or time of the year. IF we even get that amount of info.

[Draper007]

Some corporate restaurants do implement timings and machine learning not only in cooking but also in staffing. however, you need this system to be separate that will take all cooking times into account and tell the chefs to cook food in a particular order depending on cooking times and rush. it does work well but this function will have to create in the restaurant taking several factors into account especially the tools the restaurants use. every oven is different, people like to cook the same things in different ways,

[Draper007]

maybe Django backend would be better for a machine learning platform use python to create and manipulate models

[jorgemndoza]

We need a process in place to handle cancellations. This issue is most relevant to figure out internally who is going to pay for the order, delivery, etc.
User wants to cancel an order?
Driver dropped order?
Driver never delivered the order?
Restaurant ran out of food?

[FilledStacks]

@jorgemndoza yes we do. At the moment we're only considering the user cancelling their order, we'll add the status's for those states but most likely will only focus on the functionality for it far down the line. Like at the point where the driver app is developed.

[Vaytrex]

Do we gonna implement this using firebase or firestore?
Or we try to look for some opensource backend like appwrite.io since our project is opensource. We go fully opensource

[joekendal]

Firestore is the database used in Firebase.

[joekendal]

Ask yourself what's the advantage of provisioning and scaling your own infrastructure? All the code is open source so the only benefit of having an open source database is if you want to move your database. Firestore is basically like any other NoSQL database like MongoDB. Changing it at a later date is relatively easy but I doubt you'll want to manage your own servers.

[FilledStacks]

@Vaytrex the answer by @joekendal is in line with what I'm thinking. I don't want to manage servers. We'll stick with Firebase to get us rolling and building this product. It'll also be a good exercise in abstraction that we can practice in the clients. Like @joekendal mentioned in some of the discussions above, he's had the experience of Firebase not scaling as well as AWS. I think it'll scale enough, but given that his experience exists we'll build the clients in a way that it's abstracted from Firebase at the core of the code. Since the cloud functions will be written in TypeScript it'll work as an AWS lambda or an AzureFunction if those work better.

But for now it's going to be Firebase for the backend

[bawantha]

Hi may be this is stupid to ask, app like ride-sharing, delivery heavily depends on streaming platform like Kafka, what is the possibility of using that kind of streaming platform in this app.

[joekendal]

This isn't a stupid question

[FilledStacks]

@bawantha I want to reply before reading about it. I've never heard of Kafka. If this is for real time feedback like driver location tracking we're going to be using the realtime database for that and not firestore. I'll quickly check out Kafka and then let you know my gut reaction to us it.

[FilledStacks]

@bawantha this part stood out "high-performance data pipelines, streaming analytics, data integration, and mission-critical applications" - It sounds a lot like it's a great tool for something like a trading platform, live stats tracking, an analytics product etc. We don't need that kind of horsepower for a food delivery service 😄 The only real-time things we have is the order status and the driver location tracking.

[jgervin]

I assume you are storing historical lat,long for the delivery? We, at Burro, store it in an entity called a “run”. We save it on the drivers phone every 3 seconds and send up the dataset at arrival milestone in case the cell connection fails.

This allows driver to capture for tax and other purposes when using personal car.

[joekendal]

I would not recommend storing a lay and long. Instead use a geohash. This will enable you to make queries faster

[jgervin]

It is being stored that way. The point I was getting at was that you should store the drivers “lat and long” by whatever means necessary (backpack, milk jug, canteen, etc..) to have a history for the driver's use later. Also, useful later, as a company might pivot, business model change, or tax law changes. And important for big data and exit due diligence.

[FilledStacks]

@jgervin I did not think about that. Wouldn't it be just as useful to store the stats about the trip instead of the actual lat-lng for the trip over time? I'm just asking because I wasn't going to store the lat-lng over time and just store the distance travelled, time taken, average speed, time of trip, etc.

[jithin-joyc]

How do you plan to implement the real-time feature in 'Box-Out' by cloud function?
is cloud function response real-time?

[FilledStacks]

@jithin-joyc no. I'll listen to the document that I want the real-time updates from.

[jithin-joyc]

@FilledStacks caching will be handled separately?

[FilledStacks]

@jithin-joyc caching for what? I don't see the relation between that and the real-time functionality.

[jithin-joyc]

The Firebase SDKs provide a local persistence layer (cache) that stores query results for future use.
If you make the request via Cloud Functions, there is absolutely no client-side caching done by default.
If you want to cache the results, you’ll have to do that on the client, using some mechanism you choose.

[FilledStacks]

@jithin-joyc I know that but that has nothing to do with listening to real-time updates on a firestore document so I'm just confused as to what you're actually asking. The firestore SDK has 28 days local caching if that's what you're talking about. This is to avoid making a read request when you ask for the same data and nothing has changed. So I won't be doing any caching for the firestore listening because it always has to give back the latest document information at any point.

[taufeeq849]

Let's discuss the functionality of the restaurant manager and restaurant application.

As I understand it, the above two pieces of software will fundamentally enable resturants to upload products and receive orders.

However, is the process too dependant on manual input? I ask this for a few reasons:

Firstly, considering restaurants within the context of SA, they already have food delivery applications setups such as Mr D and Uber Eats. Each time they get an order, a restaurant worker still has to manually input the order into their POS system. This may lead to wrongful input into the POS system.

Secondly, the uploading of products is manual as well. Restaurants are required to upload their products to the restaurant manager application. As a result, the products displayed to the end-user may no longer be available at the restaurant since the restaurant may have forgotten to update them.

With each new 'service' that is proposed to a restaurant - the restaurant owner has to evaluate how much labour is needed in order to get the service functioning.

Therefore, should the 'food delivery service' integrate directly with the POS system? Although this may reduce manual input and labour - one has to consider whether the POS system in the restaurant enables this form of integration. Certain POS systems have API's that services can tap into but this will not be universal.

Having spoken to a few restaurant owners, they are all for new ways to get orders. But, as mentioned before, they are concerned with the added manual labour that occurs when a new 'service ' is added.

Consequently, in an ideal world, the POS system integration sounds great. But there are a few too many logistical implications. What do you guys think about this?

[FilledStacks]

@taufeeq849 Yes, The process is very dependent on manual input at the beginning. Restaurants want to control their menu's in some cases so I'll rather start out manual then transform into an automatic setup when the time comes. Restaurants that have their menu's digitized can even be scraped or imported if they have the data available.

That would only be built and thought of if there's a product running and restaurants complain about the functionality or require it before signing up.

Most of the POS systems don't work very well. If there is a common one then it'll be quite easy to add. We'll simply add another step before taking the payment to check with a POS integration service if the order can be placed and continue if it's been accepted, we'll bail on the checkout if it's denied.

The restaurants definitely don't want the manual labour, I've seen the same complaint for too many apps to get orders from, but at the end of the day they'll either have to be up to date with modern POS or just use the restaurant app. If they still take calls for orders (which is one of the oldest technologies) then they should be fine with taking it from a screen that shows them exactly what the user ordered. There's less chance of error when reading than hearing over the phone.

If the time comes where the app is up and running and it's worth adding POS integrations then we'll devise a nicely abstracted approach that will automatically integrate with a POS and allow for POS integration to be written in a standalone manner. This shouldn't be too hard to setup, then we can simply all write integrations for different systems and open source those files to hopefully cover the most popular POS's.

[DaylinBehr]

Hi Dane

Absolutely love your stuff, you're the reason I have begun my flutter development journey while still in second-year software development studies.

I have a quick question about the actual HTTPS requests and responses. Have you thought about the data compression to be used? I have not seen it mentioned in the docs but apologies if it has and I've missed it.

I believe that GZIP compression could be quite beneficial to make things even more efficient especially in the case of using express as our server architecture.

What are your thoughts?

[FilledStacks]

Hey @DaylinBehr That's awesome. I'm happy you're getting started so early.

I have not thought about compression.

If it's required to solve a problem that might appear in the future then we'll implement it that time. I don't know any of the backend system development practices so I'm doing what's needed to solve the problems we have now and then if needed we'll improve for future potential problems.

At the moment I haven't thought of compression but if it's something that's required we'll most likely use the most recommended which is GZIP I think.

Hope that answers your question.

[DaylinBehr]

Hey @DaylinBehr That's awesome. I'm happy you're getting started so early.

I have not thought about compression.

If it's required to solve a problem that might appear in the future then we'll implement it that time. I don't know any of the backend system development practices so I'm doing what's needed to solve the problems we have now and then if needed we'll improve for future potential problems.

At the moment I haven't thought of compression but if it's something that's required we'll most likely use the most recommended which is GZIP I think.

Hope that answers your question.

Yes you have! Thanks. From my experience and as I understand it the GZIP compression can be thought of more as an optimization, so not necessary however it does have a performance benefit with regards to the processing of requests and sending of responses.

I suppose this would be something only to be looked at once theres a functioning backend interacting with the client side applications and vice versa.

Probably getting ahead of things mentioning it now😅. In any case the idea is at least here.

Looking forward to the next phase of the project!

Cheers.

[manafire]

How's caching handled? In the thread above it mentions Firebase automatically caches for 28 days but wondering if this applies to Cloud Functions (/Hosting?) as well? Looking at say, the function for fetching orders - how would this be invalidated for new orders? Perhaps with a reactive hook? I know Cloud Functions take a little longer than straight calls to Firebase but do help cut the reads down. Would love to know a little more about how this all works.

Loving the videos, thank you so much! Switched over to Stacked 8 months ago and never looked back!

[FilledStacks]

Hey @manafire thank you for following along and using stacked in your own products! That's very cool.

We won't have any caching, the firebase functions will not be used for any CRUD operations. The apps will connect directly to firestore for that. The only kind of CRUD will be side effects that the RESTful functions produce when performing their action.

Other than that we'll never use it for any kind of reading the database. Well let me say, at this point we don't have any plans to read the DB through a firebase function. We'll talk directly to firestore for that.