Skip to content
Permalink
master
Switch branches/tags
Go to file
 
 
Cannot retrieve contributors at this time
6314 lines (4161 sloc) 260 KB

Post #100DaysOfCode Log - Dashiell Bark-Huss

I did a "365 days of code challenge" in 2019. I logged my progress everyday: Day 1-100, Day 101-200, Day 201-365. I continued to add to this log when I wanted to save notes in 2020.

You can see my new 2021 updates here

Tuesday 12/29/20 - Friday 1/1/21

WishTender Code Notes/Resources:


Here are some resources on debugging and optimizing react. I'm experiencing too many re-renders so I need to fix that.

How to identify and resolve wasted renders in React -Nayeem Reza

Whenever there is a change in data, React uses the component functions to re-render the UI, but only virtually Building a React app where the differentiating algorithm fails to reconcile effectively, causing the entire app to be rendered repeatedly which is actually causing wasted renders and that can result in a frustratingly slow experience.

React Wasted Renders

Learning React Developer Tools Chrome Extension

How to use the React Profiler to find and fix Performance Problems

Run Code in React Before Render:

React does not wait to render. Ever.

React will gladly kick off an asynchronous data fetch in the background, but then it will immediately proceed with rendering – whether the data has loaded or not. (and you can be almost certain that it will not have loaded yet)

What are Pure Components in React

Pure Components in React are the components which do not re-renders when the value of state and props has been updated with the same values.

...If the previous state and props data is the same as the next props or state, the component is not Re-rendered.

Note: The State and Props are Shallow Compared

...Use Pure Components, in the case when the props and state changes are made to primitive type variable, state and props changes to reference variable may lead to incorrect results and inconsistent rendering

How Can We Resolve the Shallow Comparison Problem? The simple answer is to work with immutable data...We can resolve this by creating a new instance of the state variable if it represents the reference type.

Mayank Gupta @mayank

Q: How to signal to React that a functional component is "pure", as an equivalent of React.PureComponent for component classes?

A:

const C = React.memo((props) => {
  return <var>{props.n}</var>;
});

source

prevent inline styles from causing re-renders. style objects are objects and so reference types.

const PureComponent = React.memo(Component)
...
const mainStyle = React.userMemo(()=>({backgroundColor: 'red'}))
...
return(<PureComponent style = {mainStyle}>)

source

React Class features vs. Hooks equivalents:

Comparing with the previous state\props

const Counter = (props) => {
  const [count, setCount] = useState(0);

  const prevCountRef = useRef();
  useEffect(() => {
    prevCountRef.current = count;
  });
  const prevCount = prevCountRef.current;

  return (
    <h1>
      Now: {count}, before: {prevCount}
    </h1>
  );
};

ShouldComponentUpdate

//Class
shouldComponentUpdate(nextProps) {
    return nextProps.count !== this.props.count
}

//memo
import React, { memo } from 'react';

const MyComponent = memo(
    _MyComponent,
    // Notice condition is inversed from shouldComponentUpdate
    (prevProps, nextProps) => nextProps.count === prevProps.count
)

How to debug unnecessary rerenders in React


Sunday 12/27/20 - Tuesday 12/29/20

WishTender Code Notes:


I posted a question on stackoverflow: When to ask the client their preferred currency if no country code in accept-language header?

Eventually, I answered the question myself based on a conversation I had on twitter. I might go back and update my answer, because I've realized more since I posted that answer.

False Assumptions in Questions

People weren't giving me a straight answer. I think my question is confusing. I made a lot of assumptions that aren't true.

This is one of the difficulties that pop up with self-teaching. It's really hard to form a good question because you have so many holes in your knowledge. And you also possibly have some bad assumptions about the solution.

Ultimately, I had to talk out the issue with @AndyPalmer on twitter to fully understand my blind spots. I don't know Andy personally. Andy is just a nice guy who answered my tweet and then answered all the questions I threw at him.

Here were my main misconceptions:

  • My definition of the term "middleware" was incorrect
  • I thought cookies need to always be set on the backend in some "middleware magic". They do not. They can be set using specific route requests and on the front end.
  • I was misunderstanding the jobs of the frontend, backend, and middleware.

What I thought middleware meant

I had heard middleware described as any function that gets passed the request. Like in express where you pass (req,res,next).

But I'm now hearing that middleware are these same functions, but only when every request goes through them- what I was calling "general middleware". This is like when you put app.use(somefunction) in the server page of your express app. Every request goes through it. If there's a function that is only hit on some specific routes, that can't be middleware according to this definition.

I think this second definition makes sense. Because in this second definition, the backend is separate from the middleware. All request go through the middleware. Then the backend deals with the aftermath.

Cookies don't need to be set on the backend

I thought cookies need to always be set on the backend because the only other experience I had with cookies was using express-session where the cookie is set in the backend middleware. It was just an assumption I made based on how I'd seen cookies used before.

Backend, Frontend, Middleware

Middlewares are for infrastructure concerns. The backend and frontend are where application/business logic go.

The "application" is the code that's concerned with solving your particular problem, as opposed to more generic concerns.

So, if you're building a shop, things to do with currency, shopping carts, payments etc (even if you use libraries) are your application.

If you were building a physical shop instead of a website, you'd still have to handle those things.

Things like HTTP requests, cookies, sessions, databases etc., are infrastructure; they're only important because you're building it online.

Having a separation between your application and your infrastructure means your application contains all your business rules (the why), while your infrastructure contains most of the how.

So, multi-currency to support international customers is a why, parsing the locale is a how


Twitter Conversation With @AndyPalmer:


Dashie: Anyone able to help me? Seems like an issue may people would have solved but I'm confused about what to do here.

https://stackoverflow.com/questions/65452367/when-to-ask-the-client-their-preferred-currency-if-no-country-code-in-accept-la

#codenewbie #DEVCommunity #CodeNewbies

#HelpMeCode

Andy: You'll get a few different answers. Using locale to guess country is quite clever, and saves a GeoIP call, but I'd use that as a hint and get the user to select their country/currency themselves.

E.g. I'm en_GB locale, with an Australian IP, a US keyboard, and Dvorak layout.

Thinking about why I'd recommend that, I guess my principle is it's ok to use middleware to expose things we know (such as the locale), but business/user decisions should happen in the application.

Dashie: lol I'm worried about clients like you! jk. Your suggestion is what I want to do. But how do you implement it? since cookies generally aren't set in a specific "/cookies" route, but rather any request. If it was a "/cookies" route, I'd send a res telling the client what I need.

Andy: The absence of a cookie is information. So if no country/currency cookie is set, you still need to ask the user. The application can query whether the cookie is set and decide how to proceed.

Fairly easy with JS, but might need a template that gets rendered by backend for NoJS

In both the backend conditional rendering, and the client side JS, it's the application making the decision about what no country/currency means, not the middleware (which should mainly be concerned with the HTTP infrastructure)


Side thread about what "application" means here. This thread started the next day.

Dashie: when you say "application" here does that mean either the frontend or backend but just not the middleware.

Andy: Yes. The "application" is the code that's concerned with solving your particular problem, as opposed to more generic concerns.
So, if you're building a shop, things to do with currency, shopping carts, payments etc (even if you use libraries) are your application.

If you were building a physical shop instead of a website, you'd still have to handle those things.

Things like HTTP requests, cookies, sessions, databases etc., are infrastructure; they're only important because you're building it online.

Having a separation between your application and your infrastructure means your application contains all your business rules (the why), while your infrastructure contains most of the how.

So, multi-currency to support international customers is a why, parsing the locale is a how

There are always subtleties, and this is a good example where you could make a case that multi-currency is infrastructure or locales are application.
But, if you use the "building for real life" heuristic, if someone came into the shop, you'd ask them what currency they wanted

But you probably wouldn't ask them what country they were from, because while that might give you a hint as to the currency they'll choose, the ultimate choice is the currency they want to use, not the currency of their country of origin.

Therefore, currency choice belongs in your application code (it's a first class domain object), but locale doesn't (it's a supporting role)

We're getting a bit away from your original question now though. Have you read anything about Domain-Driven Design (DDD)?


Original thread continued

Dashie: Oh right, so if no currency is set, the frontend can ask the user. But it would have to be at some appropriate time, to give time for the backend to try to set the cookie first. Maybe there's some way for the frontend to listen to incoming cookies and see if countryCode is null.

Andy: Are you trying to avoid setting any state from the client?

If not, then in a JS-on environment, I'd use something like fetch to "/choose_currency?country=US&currency=USD"
In a JS-off enviroment, I'd use Post-Redirect-Get

Dashie: Hmm I'm not sure I understand. In your example, the country is already known. But I'm looking to figure out the step before that. How do I trigger the app to ask the client for the country code/currency if the backend cannot detect it from the accept language headers.

Andy: If the cookie isn't set, you render something to the client that allows it to be set; this can be either of the solutions mentioned.
You don't set the cookie in middleware because it's an application/business logic concern, not an infrastructure concern

So, you can use the accept-language headers to provide a hint to the client (e.g. it looks like you're in the US, would you like to use USD?) but the decision still comes from the client

Dashie: Oh I hadn't heard that it shouldn't be set on the server. A lot of info I read said to have the server to set it from the set-cookie headers. There's a way to access the client's language and country prefs from the client? I was doing this from the request headers on the server.

Andy: You (usually) set it on the server. But you (usually) do it as part of an explicit client request, not by middleware magic.

E.g. currency choice is a user choice, so is set by the user. OTOH, which load balanced server you are using might be set by middleware (infrastructure)

Dashie: OK so are you saying do something like a make a specific route "/getlocale" request to the server to get the language headers, then send the country code back to the client, then handle that response by popping up a dialogue ex, if "US" then 'Would you like to use "USD"?'

Andy: Not quite. You can use middleware to annotate the request with what you know (e.g. locale)
You also know that a currency has either been chosen or not (by the cookie)
If it hasn't, then you render something that enables the user to make their choice (e.g. a form or a modal)

Submitting that form sets the cookie, and next time through, the render doesn't happen because the cookie is set.
The route should be something meaningful to the application (e.g. choose_currency), not the implementation (e.g. NOT set_cookie)

Think of observing the client requests in the network pane of the browser. It's going to be a lot easier to work out what's happening if you see:
choose_currency
login
add_to_cart

Than if you see:
set_cookie
set_cookie
set_cookie

Andy: So, say you're using handlebars, your server-side template might be:

{{!currency}}
<form>
  <select name="currency">
    {{currencies_ordered_by_locale_logic}}
    <option ...>{{/currencies_ordered...}</option>
  </select>
</form>
{{/currency}}

which only renders if currency isn't set

Dashie: Ok so like this?- a route '/choose_currency' where a middleware detects the locale , then the server sees if the there's a currency cookie. If no, the response triggers the client to render a modal to ask the user for their pref currency based on locale hints

Andy: A middleware gets hit on every request, which is why they're for infrastructure things. Use middleware to annotate a request with infrastructure things that the application doesn't need to do.
E.g. a middleware to extract country and language from the accept-language headers

The server-side template might live in your layout, so it gets rendered for every page until the client makes a decision

You could use middleware to annotate the request with the selected currency, but it feels a bit application-specific.

Generally, you'd probably store the currency (and other user data) in a session store, and the cookie would identify the session.

Then the session store middleware would retrieve and annotate the request with the user data. That's the usual compromise; the middleware only knows about session state, it's not making business decisions.

Your application asks for request.session.currency

See npm: express-session

Dashie: oh, I've been using the term middleware wrong- to mean any function in the app that gets passed the request. I used the term 'general middleware' to more specifically describe a middleware that gets hit on every request. So my apologies if I said anything confusing here

Andy: Express doesn't particularly help with its documentation; the distinction is much clearer in, for example, Ruby with Rack.

If you think of it in terms of a block diagram, it looks something like:

Web server -> middleware -> application

Middleware is for doing extra processing (for example, automatically decoding JSON in the body of a POST), but ideally it shouldn't know anything about the application.

Similarly, the app shouldn't care about how the JSON got there, it should just do application things with it.

So, extracting locale is a good use of middleware because it isolates your app from knowing how it got the locale. But setting a currency cookie is a bad use of middleware because it means your infrastructure knows about application logic (no separation of concerns)

And although your application code has the same "shape" as a middleware, it's in a different conceptual area, so it would probably be better called a controller (as in Model-View-Controller)


Wednesday 12/24/20

WishTender Reflections:


Internationalization for Currency

I'm starting to implement the currency internationalization for my app. There will be a lot of different currency's used by wishers and gifters.

My signed up beta users are Canadian, British, or American.

I didn't find a lot of information on internationalization especially for currency. I found a bit for language which, though cumbersome, seems more straight forward.

I started a sample project to share my solution. Right now it just determines the preferred locale and sets it on the cookie. More to come.

My Sample Internationalization Project


Tuesday 12/15/20

WishTender Reflections:


Christmas Deadline Not Met

At this point I'm not going to finish building beta by Christmas. But that's ok. I hope people aren't too disappointed. I didn't make any promises, but I had been hoping to get WishTender out before Christmas so people could get their Christmas Gifts.

But things are going well. As you can see, a lot of the wishlist is functional. Backend and frontend connected. And it's pretty! But still more work do to. Especially with the frontend of stripe, internationalization, login, logout, sign up, managing purchased gifts, and connecting it all to the backend.

This is Hard But I Don't Care.

I keep running into problems I don't understand. But I keep figuring them out. Faster and faster. That makes me calmer every time I run into a problem the next time. So I have been really calm and confident lately.

Long Term Projects Build Confidence

Omara @omarakhaddaj once said to me, You develop self-confidence by overcoming challenges and creating small wins.

Working on a big project like WishTender gave me many opportunities for small wins. It doesn't matter how long it takes me, I am committed to finishing.

With other projects, I was not committed to finish. So when I ran into a wall, I would just start a new project. That's not terribly bad; all of those other projects were just for learning. But moving from project to project doesn't build the confidence of a long term project.

WishTender is for real. So I must finish it. This forces me to see the problems through to the end. I must conquer every challenge no matter what. Overtime, this has built a lot of confidence and calm in me. It's a powerful feeling. All it took was patience and persistence.

Confidence Cycle

The confidence itself brings about more wins. In my past it wasn't my skills that held me back, but my confidence. It took me a while to realize I don't need to rely on my skills and knowledge; It's necessary to make attempts at projects outside of my skill level.

You're Not Ready is the default

In the past I thought, "I'm not ready" far too often. Now I see things differently. Not being ready is the default. If you're not ready, you're ready.

Maybe thinking about it too much is the problem. Feeling unskilled is almost egotistical; it's too much focus on the self. It shouldn't be about about me me me. It's about what can I do to make this happen? You are a conduit. As Tim Gunn says, "Make it work, designers!"

Because I see things differently, I don't shy away anymore from projects outside my skill level. Which leads to more wins and more confidence.

When To Hire?

I wonder when will be the right time to start hiring people to work on WishTender with me? Is it when WishTender starts making money?

A few people have reached out to me asking to work for free. But I turned them away because I didn't think it would benefit me or them at the time, since I was also learning how to do every thing on my own. I do feel like I'm getting closer to a point where I could work with other people, but I would need some guidance.

Thanks for reading! Let me know if there's anything else you're interested in hearing about next update.


Monday 11/30/20 - Tuesday 12/1/20

WishTender Reflections:


Asking Lots of Questions

I recently heard someone talk about the "myth of individualism": The myth that we can and should do everything by ourselves.

This is something I'm guilty of. I believe it holds me back.

In response, I've been asking for help more often. Particularly on forums.

Outside of personal gains, there's another reason to ask questions. I realized that if I have a problem, asking it might help others who have the same question. So it's my responsibility to ask the question.

Most of my questions go on StackOverflow, Youtube comments, the comments on StackOverflow answers, and chat groups. The only active dev chat I'm on is the 100daysofcode slack. I'd like to find more. Some facebook groups have been helpful too.

You can see some of the questions I've been asking here.

Be Bold And Ask Questions

Sometimes we believe only stupid or annoying people ask questions. Stupid because they can't figure out answer. Or annoying because they refuse to. So we avoid asking questions to avoid being seen as stupid or annoying. But I think bold people ask questions. So I'm going to be more bold.

Bold and persistent. Who cares if someone thinks I'm stupid? As my Shakespeare acting teacher always said, "Welcome to my process, fuck you very much."

What I Do When I Run Into A Wall

CORS Cookies and Sessions Oh my

I recently found I had a problem where my app creates a new session for every request coming from the frontend, even though all the requests are coming from one browser. That should just be one session!

I've been investigating this problem and I believe it has something to do with either cors, cookies, or sessions. Concepts I don't a lot about.

Stoicism When Facing My Own Incompetence

When I run into a wall like this- where I have no clue where else to turn, I spend some time learning about the concepts that I believe will help me get closer to the answer.

These roadblocks used to get me down. I'd think, "Who the hell do I think I am to try to make a full app?" But then I'd solve my problem 24 hours later and feel confident again.

So I've learned not to let these setbacks get to me because I turn them around quickly. I'm pretty confident I will figure my problem out in the next day. And if it takes me a week to understand the concepts I need to learn, that's ok too.

Why Don't I Just Hire a Dude in India To Build My App?

My entrepreneur friend asked me this question the other day. He said I could pay someone $200 bucks to make my whole app. I don't know if that's exactly true, but there's truth to it. I could outsource this project.

Outside from the horror stories I've heard of outsourcing full applications for cheap, there are other reasons I want to build WishTender myself.

The Value of Doing It Myself

I really want to understand how building an app works before I pay someone to do it. Learning software engineering will put me in a better position in the future. I'll know what to look for when hiring people. I'll know how to talk and think about tech products. And I can prototype WishTender features or completely knew project ideas.

Software engineering is no doubt a beneficial skill to add to my talent stack. And since I enjoy it and am good at learning it, why not lean into that skill? While I want to launch WishTender before Christmas, it's not the end of the world if I don't.

Not only is it better for me, but I believe it's better for the health and longevity of WishTender and it's future customers if this app is built on a solid foundation.

Don't Fail Faster Than You Need to

People say fail fast. But WishTender is also a learning experience, so there's no need to go faster than necessary.

If I pay someone 10k to build WishTender -the cheapest real life figure I've come across- and WishTender fails, what am I left with? A small learning experience about outsourcing that cost me $10k?

There's such a thing as failing unnecessarily fast. If I build WishTender myself and it fails- I learned how to build an app and I'm in a much better position to pivot and ultimately serve my customers better.


Sunday 11/29/20

WishTender Update:


PUT vs PATCH


Tuesday 11/24/20

WishTender Update:


Designing URI for current logged in user in REST applications

  • With symbolic link

    This solution has a symbolic link for the ID of the user.

    Example:

    • Confluence REST API: GET /user/current

Monday 11/23/20

WishTender Update:


WishTender was Banned from Facebook

A while back, I posted about my meeting with the CEO of Deskpass. He suggested WishTender be more upfront about it's primary niche- sex workers. I thought this was a good suggestion, but I wondered why OnlyFans doesn't play up their sex worker niche. Similar to OnlyFans, WishTender is not exclusively for sex workers. But sex workers frequently get gifts from their fans, so they are a huge market for us.

However, being upfront about the sex worker aspect is already proving difficult. Last week, Facebook took our page down because we included the term "SW friendly". We had no other sex related content, just that acronym.

It's very frustrating that my business is being affected by something out of my control. But a founder doesn't let setbacks get to them. A founder keeps going. A found is anti fragile- we grow stronger in the face of obstacles. So I am told, anyways!

My team went through the minimal recourse available for reverting the ban. If that doesn't work, we'll create a new page and leave out 'SW friendly'. Now we have to rethink how upfront we want to be about our target market.

I will remind you that when I say team, I mean myself, my spouse, and sometimes my sister. WishTender is bootstrapped and I'm coding the whole thing alone. I often wonder is this is stupid of me.

Frontend

I've been working on the frontend more, and connecting it to the backend.

This profile section was inspire by

  • Blueprint Registry
  • Twitter
  • Facebook (yes, the haters I mentioned above)
  • Instagram

This profile component is on this github: Frontend

The companion backend: Backend

Oh there might be a clickable overlay on the profile photo that errors- I forgot to delete it. So yeah this isn't totally cleaned up yet.

Focusing: Launch by Xmas?

I blocked almost all notifications from my phone. I am trying to focus so I can launch before xmas (we'll see).

Social media is on the back burner, but I come on every once in a while to check messages and make a post. I love using social media to connect, but building WishTender is my priority.

I'm not consuming news, nor making any additional plans to hang out besides Shabbos, Hanukkah, and Thanksgiving. Even those are taking up a lot of time.

I'm watching very little media. Only an occasional show at the end of the day with my spouse- while I code.

I take breaks to work out, cook and eat meals, and take walks with my spouse.


Tuesday 11/19/20

WishTender notes/resources:


Thinking in React

So when we have a lifted state, aka when the state is in the parent, the state is changed by an action on the child such as onsubmit, we was to pass in a function for on submit that is defined in the parent.

components should only update their own state

My stack over flow question


Tuesday 11/17/20

WishTender notes/resources:


Thinking in React

Let’s go through each one and figure out which one is state. Ask three questions about each piece of data:

  1. Is it passed in from a parent via props? If so, it probably isn’t state.

  2. Does it remain unchanged over time? If so, it probably isn’t state.

  3. Can you compute it based on any other state or props in your component? If so, it isn’t state.

Remember: React is all about one-way data flow down the component hierarchy. It may not be immediately clear which component should own what state. This is often the most challenging part for newcomers to understand, so follow these steps to figure it out:

For each piece of state in your application:

  • Identify every component that renders something based on that state.
  • Find a common owner component (a single component above all the components that need the state in the hierarchy).
  • Either the common owner or another component higher up in the hierarchy should own the state.
  • If you can’t find a component where it makes sense to own the state, create a new component solely for holding the state and add it somewhere in the hierarchy above the common owner component.

React Docs: AJAX and APIs

Where in the component lifecycle should I make an AJAX call?

You should populate data with AJAX calls in the componentDidMount lifecycle method. This is so you can use setState to update your component when the data is retrieved.

But what if the Ajax call is a post?

React + Fetch - HTTP POST Request Examples -> Here are a lot of post examples and the request is also called in component did mount. But I don't understand- aren't we calling post request onclick or submit? That would be after the component already rendered. Why is it being called in the beginning of a component being rendered?

I refactored the profile section and added components for the user to edit their page when logged in. That woman in the cover photo isn't me by the way but that's me in the profile picture with my mom. The women in the cover is a mystery woman I call Britney. This one's for you Britney.


Monday 11/16/20

WishTender notes/resources:


Styling React


Sunday 11/15/20

WishTender notes/resources:

Material Ui, better imports

Video on Material UI theming.


Friday 11/13/20

WishTender notes/resources:

React: Lifting Up the State

Just spent several hours trying to debug why my component stopped working all of the sudden. I finally realized it was because I wrapped the component in a div originally. But then I changed it to a form which changes the default for all the buttons. So I have to add e.preventDefault(). I needed to write that down so I never forget to add preventDefault! It was confusing because the button onclick handlers still worked half way. But as soon as the page refeshed- the default action for form buttons- the function stopped working. I also hadn't git committed it so i couldn't go back in time and eee why it was working before.


Wednesday part 2 11/12/20

WishTender notes:

Should react components have widths? Or should they be put in containers with widths?

Wednesday 11/11/20

WishTender notes:

Crop and Send User Images to the Server

I made a sample project that shows how to crop and upload an image to your server in React and express.js. I'm using this to allow users to upload profile and banner images.

Demo Video

IMAGE ALT TEXT HERE

Code


Monday 11/10/20

WishTender notes:


Utility Functions

Put both functions in an external JavaScript file called utils.js within the same directory as the component. Since the functions will be exported as modules, you need to prepend it with the export keyword. utils.js will look like this:

-source

File Structure for React Applications Created with create-react-app

Sunday 11/8/20

WishTender notes:

After an api post request comes back successful, you can use redirect to go to a new page.

A use case for this: After my users fill out the sign up form, the onSubmit function for that form sends a post request to my api. If the request returns a successful response, I want to redirect the user to a form to get their profile and wishlist information. I don't want to redirect them when they click submit because the sign up might not be successful, for example if someone else has the same email.

<Redirect>

Rendering a <Redirect> will navigate to a new location. The new location will override the current location in the history stack, like server-side redirects (HTTP 3xx) do.

<Route exact path="/">
  {loggedIn ? <Redirect to="/dashboard" /> : ><PublicHomePage />}
</Route>

React Router Docs: Redirect


Saturday 11/7/20

WishTender notes:

Aakash: You should never enable cors.

Shubham: I wouldn't agree with your comment completely, you need to enable cors when you need to serve data to a frontend that is on a different domain, however, I would agree that you might want to put a filter on domains that you trust

-source

Adding a Proxy when using webpack

To tell the development server to proxy any unknown requests to your API server in development, add a proxy field to your package.json, for example:

"proxy": "http://localhost:4000",

This way, when you fetch('/api/todos') in development, the development server will recognize that it’s not a static asset, and will proxy your request to http://localhost:4000/api/todos

-source

I was wondering why adding a proxy to my package.json file in the frontend didn't redirect fetch request to my backend server. I found this answer. It was because I have webpack set up (I didn't know I did? I think it comes with create-react-app). Webpack


Friday 11/6/20

WishTender notes:

Resources on React

React Forms Form Hook

React Docs React Context

useContext() + useReducer() = Magic?

Good Explanation on what problem redux and React Context solves.


Thursday part two, 11/5/20

WishTender notes:

There's an extension for VSCode called VS Code ES7 React/Redux/React-Native/JS snippets. It's like autocomplete for React snippets. Like Emmet for React.

React & Express Starter Pack For Full Stack Development

Put CSS in Component Folder

I like this file structure for React. Put the css for each component in the component file. My css right now is in one place and so crazy messy.

How to run The react server and the Express Node Server At the Same time: Video


Thursday, 11/5/20

WishTender notes:

Chat With Deskpass Founder, Sam Rosen

I got in touch with the founder of Deskpass, Sam Rosen. Our moms are long time friends.

It was great to just hear his thoughts and get encouragement. I was nervous reaching out, but I'm glad I did. Sam was extremely generous and helpful.

Sam's Advice:

Be Persistent

Sam said that the biggest determinant in being a successful entrepreneur is persistance. There will be times when it's hard. Times when it's easy. Persist through it all.

Be Obvious In Your Messaging

Sam said when he looked at WishTender's site and the twitter, it took a while for him to realized the main use case is for sex workers/cam girls/OnlyFans.

The only mention here of sex workers is "SW Friendly". Most people don't know what SW friendly stands means. And we also only added that in or bio recently and have no mention of it on our site.

The average person doesn't take the time to understand your product. So be obvious.

Since WishTender has a very good product market fit with sex workers, we should go after that niche. Which was my plan. Still I thought we could leave it more open ended. But as Sam put it, most businesses have a digestion problem not a starvation problem- they try to do everything instead of focusing on what works.

I haven't been obvious about the sex work connection because I thought it would turn some people away. Maybe even offend my Twitter followers, which I probably shouldn't worry about but I do. But I think Sam is right.

Sam’s biggest suggestion: Focus on the sex worker niche. Be more obvious about that intended use case. Call it a wishlist site for OnlyFans/sex workers.

Trust Symbols

WishTender gives wishers the gift money directly. It's up to wishers to then purchase the gift.

But how does this make the gifters feel? Might they worry that the wisher won't spend it on the intended gift?

Anytime you want someone to pay, you need to validate them. Show there is trust. No one wants to feel like they might be scammed.

I need to include validation to the gifter that they're getting what they want though trust symbols. Trust symbols could be a rating system on wishers profile, for example. Or display how many times the wishers sent "thank you" messages.

I'm trying to think of a way to do this that feels like it fits in on WishTender. Requiring ratings for a wisher doesn't feel right. Maybe I'm wrong, but it doesn't feel like it's a gift if you get rated on how you receive it.

Could there be an optional page for wishers to post "thank you" pictures from previous gifts? Or an optional display of ratings? This might be something to market research.

Use Cameo's Messaging System As A Template

Sam says Cameo does a great job of managing communication between celebrities and fans.

If I was buying a Cameo from my favorite entrepreneur Gary Vee, I would make up all sorts of problems with my order and excuses to keep talking to him. (I love you Gary!) So cameo only let's you write a certain number of characters and messages to prevent fan girls like me from crazed nonsense.

I'm going to have similar concerns with WishTender, so Cameo's messaging system is a great model to follow.

Featured Teachers:


Wednesday, 11/3/20

WishTender notes:

Doh! Dynamic Objects In The Database Don't work

I watched this tutorial a few weeks ago about making cart object for shopping. I thought I was so smart for improving the cart by making it dynamic; Instead of setting cart.totalPrice to the total price, I set it to a getter function.

// example before:
this.totalPrice = items.reduce(
      (a, item) => aliasCart.items[item].item.price * aliasCart.items[item].qty + a,
      0
    );
// example after:
this.totalPrice = get totalPrice() {
          const keys = Object.keys(this.items);
          const total = keys.reduce((a, c) => this.items[c].item.price * this.items[c].qty + a, 0);
          return total;
        },

The first one is static. If the result of this.totalPrice is 4 then it will stay 4 even if we change the price of an item. In the second example, this.totalPrice is dynamic. It will change based on the prices and quantities.

But since we store the cart in a session, it gets stringified. So it loses its dynamic functions and becomes a static object. So this was no help. I changed it back to a static function.


Sunday, 11/1/20

WishTender update:

I've been working on internationalization and finishing up stripe integration.

Currency Helper

I made a Currency Helper that integrates with an exchange rate api interface. Here's the gist. Here's the code:

var axios = require("axios");
const countryData = require("country-data");
const fx = require("money");

/**
 * Currency helper
 * @constructor
 * @param {Object} exchangeRateInterface an api interface for an exchange rate api site
 */
function CurrencyHelper(exchangeRateInterface) {
  this.exchangeRateInterface = exchangeRateInterface;
  this.countryData = countryData;
  this.fx = fx;

  /**
   * Gets the exchange rate for a currency from an api
   * @param {String} from the starting currency
   * @param {String} to the ending currency
   * @returns {Number} the exchange rate to multiply the starting price
   */
  this.getExchangeRate = async function getExchangeRate(from, to) {
    const exchangeRate = await this.exchangeRateInterface.getExchangeRate(
      from,
      to
    );

    return exchangeRate;
  };

  /**
   * Gets the exchange rates for a currencies from an api
   * @param {String} [baseCurrency='USD'] optional. default 'USD'.
   * @returns {Number} the exchange rate to multiply the starting price
   */
  this.getAllExchangeRates = async function getAllExchangeRates(
    baseCurrency = "USD"
  ) {
    const exchangeRate = await this.exchangeRateInterface.getAllExchangeRates(
      baseCurrency
    );
    return exchangeRate;
  };

  /**
   * Get rates and update this.rates
   * @param {String} [baseCurrency='USD'] optional. default 'USD'.
   * @returns {Boolean} object with updated and rates
   */
  this.getAndUpdateRates = async function getAndUpdateRates(
    baseCurrency = "USD"
  ) {
    const rates = await this.getAllExchangeRates(baseCurrency);
    this.rates = rates;
    return this.rates;
  };

  /**
   * Convert smallest unit of currency to the proper decimals
   * @param {Number} smallestUnit the amount of the smallest unit of currency (ex: USD- amount of pennies)
   * @param {String} currency the 3 letter code for the currency. ex: 'USD'
   */
  this.smallestUnitToStandard = function smallestUnitToStandard(
    smallestUnit,
    currency
  ) {
    const { currencies } = this.countryData;
    const multiplier = 10 ** -currencies[currency].decimals;
    return smallestUnit * multiplier;
  };

  /**
   * Convert smallest unit of currency to the formatted price
   * @param {Number} smallestUnit the amount of the smallest unit of currency (ex: USD- amount of pennies)
   * @param {String} languageCode ex: en-US
   * @param {String} currency the 3 letter code for the currency. ex: 'USD'
   */
  this.smallestUnitToFormatted = function smallestUnitToFormatted(
    smallestUnit,
    languageCode,
    currency
  ) {
    const standard = this.smallestUnitToStandard(smallestUnit, currency);
    const formatted = this.formatCurrency(standard, languageCode, currency);
    return formatted;
  };

  /**
   * Convert currency to the smallest unit
   * @param {*} price
   * @param {String} currency the 3 letter code for the currency. ex: 'USD'
   */
  this.priceToSmallestUnit = function priceToSmallestUnit(price, currency) {
    const { currencies } = this.countryData;
    const multiplier = 10 ** currencies[currency].decimals;
    return fx(price)._v * multiplier;
  };

  /**
   * Gives the price in the correct currency formatting
   * @param {*} price
   * @param {String} languageCode ex: en-US
   * @param {String} currency ex: USD
   *
   * @returns {String} formatted price
   */
  this.formatCurrency = function formatCurrency(price, languageCode, currency) {
    return new Intl.NumberFormat(languageCode, {
      style: "currency",
      currency,
    }).format(price);
  };

  /**
   * Convert a price to another currency
   * @param {Number} price
   * @param {String} from the starting currency
   * @param {String} to the desired currency
   */
  this.convert = function convert(price, from, to) {
    this.fx.rates = this.rates;
    return this.fx(price).from(from).to(to);
  };
}

//______________________________________

class ExchangeRateApiInterface {
  /**
   * api.exchangeratesapi.io API interface
   * @constructor
   */
  constructor() {
    this.baseURI = "https://api.exchangeratesapi.io";
  }
  /**
   * Gets the exchange rate for a currency from api.exchangeratesapi.io
   * @param {String} from the starting currency
   * @param {String} to the ending currency
   * @returns {Number} the exchange rate to multiply the starting price
   */
  async getExchangeRate(from, to) {
    const exchangeRate = await axios
      .get(`${this.baseURI}/latest?base=${from}&symbols=${to}`)
      .then((x) => {
        return x.data.rates[to];
      })
      .catch((response) => {
        throw new Error("Error: " + response.response.data.error);
      });
    return exchangeRate;
  }

  /**
   * Gets the exchange rates for a currencies from api.exchangeratesapi.io
   * @param {String} baseCurrency
   * @returns {Number} the exchange rate to multiply the starting price
   */
  async getAllExchangeRates(baseCurrency) {
    const exchangeRate = await axios
      .get(`${this.baseURI}/latest?base=${baseCurrency}`)
      .then((x) => {
        return x.data.rates;
      })
      .catch((response) => {
        throw new Error("Error: " + response.response.data.error);
      });
    return exchangeRate;
  }
}
//______________________________________

(async () => {
  //example uses
  const api = new ExchangeRateApiInterface();
  const currency = new CurrencyHelper(api);

  currency.smallestUnitToFormatted(999, "en-US", "USD"); //> $9.99

  await currency.getAllExchangeRates("USD"); //> {"CAD":1.3297999658,"HKD":7.7539750385,"ISK":140.5368439049,"PHP":48.4142588477,"DKK":6.3657035391," ...

  await currency.getExchangeRate("USD", "GBP"); //> 0.7711403659

  // update this.rates to use in convert()
  await currency.getAndUpdateRates("USD");

  const converted = currency.convert(9000, "USD", "GBP");
  const formatted = currency.formatCurrency(converted, "en-UK", "GBP"); // £6,940.26
})();

This Currency Helper helps me:

  • Convert prices between currencies
  • Update the exchange rates used because they change frequently
  • Format currency
  • Get the prices in smallest unit for each currency. Stripe takes prices in pennies.

I used the facade pattern to have the Currency Helper interact with an exchange rate API but not rely on it. This way, if I find that the exchange rate API that I'm using only updates exchange rates once a day, I can easily change out to different exchange rate API that updates every second. Usually these frequently updating apis are expensive. So the facade patterns helps me use the free API while I wait to afford the better one.

Sinon

I answered my own question on stack over flow:

How to use Sinon to overide mongo validations in tests

I just had to add

sinon.stub(Object.getPrototypeOf(Wishlist), "findOne").callsFake(() => "mock");

to my test.

This way, when WishlistItem tries to validate that the parent Wishlist exists by calling Wishlist.findOne, instead () => 'mock' is called in place of findOne. SKipping the validation that will fail unless we create a wishlist.


Wed, 10/28/20

WishTender messy notes:

Exchange Rates API

Api for exchange rates: https://api.exchangeratesapi.io/latest?base=USD

Currency Formatting in ES6 Javascript:

new Intl.NumberFormat(
  "en-UK", // locale
  { style: "currency", currency: "GBP" }
).format("9000");
// returns-> "£9,000.00"
(73.57).toLocaleString("de-DE", { style: "currency", currency: "EUR" }); // German: 73,57 €

Not sure what's the difference.

The locale is meant up of a language code and country code.

Internationalization (i18n)

Internationalization (i18n) involves adding support for different languages and countries in your app. The number 18 stands for the number of letters between the first ‘i’ and the last ‘n’.

-source

Detect location and local timezone of users in JavaScript:

fetch("https://extreme-ip-lookup.com/json/")
  .then((res) => res.json())
  .then((response) => {
    console.log("Country: ", response.country);
  })
  .catch((data, status) => {
    console.log("Request failed");
  });

This works by making a request to the https://extreme-ip-lookup.com/json/ URL from the user’s browser, so that their location is detected. This resource provides country, city, time zone, longitude, and latitude among other things.

Featured Teachers:

WishTender Update


Thursday-Tuesday, 10/22-27/20

WishTender Update

I haven't pushed this blog to remote in about since Tuesday, 10/13/20. But I have a few entries in here since then.

I didn't think anyone was reading. But I got a message asking for my next post. So here's an update.

Stripe Integration

I've been working on integrating Stripe. Stripe is a payment processing platform which you can integrate with using their API.

WishTender is technically a marketplace. Wishers post what they want-> Tenders send gift funds (or wish tender?) to the wisher so the wisher can buy the item. For this to work, I need to be able to collect money from the tenders, and send money to the wishers. Stripe has 'connected' accounts for this.

A wisher creates a connected account with Stripe through WishTender. For the connected account, Stripe collects bank account information etc from the wisher. Now when a tender tends a wish -aka: sponsors a gift- WishTender collects the money, takes a fee, and sends the rest of the funds to the wisher's connected Stripe account.

I'm really enjoying working with Stripe's API. They have 24 hour chat support which is awesome, although sometimes the support doesn't get things correct. Stripe also has great tools for testing.

Stripe Fees

I've been trying to understand exactly what fees I'll be charged by Stripe. On top of Stripes normal processing fee (2.9% +.30 cents), Stripe charges a fee for connected accounts (.25% + 25 cents + $2 monthly fee), currency conversion (1%), and transferring funds to an international connected account (1%).

Some of the details are confusing. I had to talk to Stripe support about the details. For example, you get charged for the $2 monthly fee only if the user was active that month- but it's not by calendar month. So I got the details on when exactly those fees go through. If you're interested in the details let me know and I can share them.

The reason I wanted all the fee details was to be able the pass the Stripe fees on to the tender.

I made this fees function to calculate the fees for me.

/**
 * Creates Fees object
 * @param {Int} giftPriceTotal price in pennies ($1 would be 100)
 * @param {Number} appFee percent of app takes of gift price (10% would be 10)
 * @param {Boolean} accountFeeDue Is the $2.00 account fee due?
 * @param {Boolean} internationalPresentment Is was the currency presented by stripe in a currency other than US?
 * @param {Boolean} internationalDestination Is the wisher an account outside the US?
 *
 * @returns {Object} fees object, fees in pennies most
 * important: this.stripeTotalFee & this.appFee
 */
function Fees(
  giftPriceTotal,
  appFee,
  accountFeeDue = false,
  internationalPresentment = false,
  internationalDestination = false
) {
  const roundToPenny = (pennies) => parseInt(pennies.toFixed());
  this.accountFeeDue = accountFeeDue ? 200 : 0;
  this.currencyConversionPrct = internationalPresentment ? 0.01 : 0;
  this.internationalTransferPrct = internationalDestination ? 0.01 : 0;
  this.appFee = roundToPenny(giftPriceTotal * appFee * 0.01);
  this.charge = roundToPenny(
    (giftPriceTotal + this.accountFeeDue + this.appFee + 55) /
      (1 -
        (0.0315 + this.currencyConversionPrct + this.internationalTransferPrct))
  );
  this.stripeTotalFee = roundToPenny(
    this.charge *
      (0.0315 + this.internationalTransferPrct + this.currencyConversionPrct) +
      55 +
      this.accountFeeDue
  );
  this.stripeFee = roundToPenny(this.charge * 0.029 + 30);
  this.stripeConnectedFee = roundToPenny(this.charge * 0.0025 + 25);
  this.internationalTransferFee = roundToPenny(
    this.charge * this.internationalTransferPrct
  );
  this.currencyConversionFee = roundToPenny(
    this.charge * this.currencyConversionPrct
  );
  this.stripeFeesBalanced =
    this.stripeFee +
      this.stripeConnectedFee +
      this.accountFeeDue +
      this.currencyConversionFee +
      this.internationalTransferFee ==
    this.stripeTotalFee;
  this.wishersTender = giftPriceTotal;
  this.total = this.wishersTender + this.stripeTotalFee + this.appFee;
  this.balanced = this.total == this.charge;
  if (!this.balanced || !this.stripeFeesBalanced)
    // this error handler is just to make sure I wrote this function correctly while testing the function with different values.
    throw new Error(`fees aren't balanced, refactor this function`);
  return this;
}

Exchange Rates

I want to present my international tenders with wish prices in their currency. For example, if I have a wisher in Canada with a CAD$100 purse on their wishlist, I want a British tender to see the purse in GBP in stead of CAD. Today, C$100 is worth 58.22 GBP. So, today the British tender should pay 58.22 GBP for the purse. Stripe then converts the 58.22 GBP to C$100 and puts it in the Canadian wisher's connected account.

But exchange rates change all the time. So I have to figure the exchange rate at the time the tender is browsing the wishers list.

Stripe doesn't provide you with their exchange rates so I will try using the free plan of Open Exchange Rates API to get exchange rates.

Let Me Know If You Want More

Let me know if you want more info, code samples, tutorials etc. about what I talked about today. Also, my WishTender repo is private, but I would consider sharing more of it if it would help. My Twitter DM's are always open if you have any questions. I can also meet over video chat.

Messy Notes:

Sales Tax

State sales taxes: Am I supposed to add sales tax? Is there anything in the stripe API for this?

My CBD Entrepreneur Neighbor's Answer: It's not clear what the online sales tax requirements are. Don't worry about it until you start making a good amount of money.

Node Archetecture and 3rd party apis

Use a layer for third party services Node

Use a Pub/Sub layer too

I used to really like stripe support. But they have been getting things wrong lately.

Using MongoDB in Tests

I'm been using mongoose to add documents to the database before running mocha tests. My mongoose schemas are all tied together. It isn't very modular like unit test should be.

For example, if I need to run tests on a Wishlist document, I have to create a User and an Alias too. This is because a Wishlist will fail validations if it isn't provided an existing Alias. And Alias will fail validation if it isn't provided an existing User. So I must create all of them to make a Wishlist: User->Alias->Wishlist.

This is to make sure that every document has an owner and doesn't get lost in the database.

But it isn't very test friendly.

Then I saw this code where mongoose is not being use, just MongoDb. Could I get around the mongoose schema validations by using MongoDb to create test data instead of Mongoose with this?

const { MongoClient } = require("mongodb");

describe("insert", () => {
  let connection;
  let db;

  beforeAll(async () => {
    connection = await MongoClient.connect(global.__MONGO_URI__, {
      useNewUrlParser: true,
    });
    db = await connection.db(global.__MONGO_DB_NAME__);
  });

  afterAll(async () => {
    await connection.close();
    await db.close();
  });

  it("should insert a doc into collection", async () => {
    const users = db.collection("users");

    const mockUser = { _id: "some-user-id", name: "John" };
    await users.insertOne(mockUser);

    const insertedUser = await users.findOne({ _id: "some-user-id" });
    expect(insertedUser).toEqual(mockUser);
  });
});

Source: Use jest-mongodb Preset

Seeded MongoDb in Mocha Test

I was able to add mongoDb documents without mongoose. Now I can add test data without having to worry about mongoose validations!

it("should tell if didGetOrderLast30Days", async () => {
  db = await connection.db("test");
  const orders = db.collection("orders");

  const userId = mongoose.Types.ObjectId("5f9480a69c4fcdc78d55397d");
  const userId2 = mongoose.Types.ObjectId("5f9480a69c4fcdc78d553971");
  const mockOrder1 = {
    user: userId,
    createdAt: new Date(Date.now() - 29 * 24 * 60 * 60 * 1000),
  };

  const mockOrder2 = {
    user: userId2,
    createdAt: new Date(Date.now() - 31 * 24 * 60 * 60 * 1000),
  };

  await orders.insertOne(mockOrder1);
  await orders.insertOne(mockOrder2);

  const didGetOrderRecently = await orderService.didGetOrderLast30Days(userId);
  const didNotGetOrderRecently = await orderService.didGetOrderLast30Days(
    userId2
  );

  didGetOrderRecently.should.be.equal(true);
  didNotGetOrderRecently.should.be.equal(false);
});

Accounting.js for formatting money

All of Stripe's currency conversions occur at approximately 2% [1% for US accounts] above the daily mid-market rate, which you can view on any up-to-date online conversion calculator for example this one here: https://openexchangerates.org/

-Stripe support chat


Wednesday, 10/21/20

Daily WishTender Update

Today's Progress:

  • More stripe docs and talked to support: international charges, taxes
  • Worked with the stripe api to test currency conversion and transferring funds to other countries,

Legal Implications of a Chargeback on No-Refund Policy

Stripe Pricing:

Fees by country are listed below for Express and Custom accounts. If currency conversion is required, an additional fee will apply: 1% for US accounts and 2% for non-US accounts. If transferring funds to other countries, an additional 1% cross-border transfer fee will apply.

The fee applies on transfers to international payout recipients. Notably, it does not apply to direct charges or charges with the “on_behalf_of” parameter set.

Currency Conversion: +1%

A currency conversion occurs if the presentment currency differs from the settlement currency.

The presentment currency is the currency that’s used for charges. The settlement currency is the currency that you can receive payouts in, depending on the charge type and applicable currency conversion.

source: Connect: Working with multiple currencies

Transferring Funds to Other Countries: +1%

Cross-border payouts enable you to pay sellers, freelancers, content creators, and service providers in their local currencies. You can transfer funds to connected accounts in other countries with your existing platform account and charge configuration.

Cross-border payouts only work with the recipient service agreement.

source: Cross border payouts

Transfers to recipient accounts take an extra 24 hours to become available in the connected account’s balance. ...

const account = await stripe.accounts.create({
  country: "SG",
  type: "custom",
  capabilities: {
    transfers: {
      requested: true,
    },
  },
  tos_acceptance: {
    service_agreement: "recipient",
  },
});

source: service agreement types

If you know the country and capabilities for your connected account, you can provide that information when you create the account. Connect Onboarding then collects the requirements for those capabilities ...

const account = await stripe.accounts.create({
  country: "US",
  type: "express",
  capabilities: {
    card_payments: {
      requested: true,
    },
    transfers: {
      requested: true,
    },
  },
});

sourcecreate an express account

Since the country is specified we could identify if a 1% charge needs to be added.

Onboarding express accounts outside of your platforms country

Stripe supported currencies:

If the charge currency differs from the customer’s credit card currency, the customer may be charged a foreign exchange fee by their credit card company.

Stripe best practices fruad

Stripe Tax Reporting

Stripe checkout demo


Tuesday, 10/20/20

Daily WishTender Update

Today's Progress:

  • Created a Stripe destination charge flow in a separate project folder That calculates stripe fees except international charges

Working with multiple currencies

Another way to avoid currency conversions when performing separate charges and transfers is to use the on_behalf_of parameter.

Charge Types update

Yesterday I thought I couldn't do destination charges, but now I see I can. I thought destination charges put the entire payment in to the wishers account. But that's only under some settings. Here is how to specify the transfer amount and keep your app fee, instead of transferring the entire amount and taking an app fee after.


Monday, 10/19/20

Daily WishTender Update

Today's Progress:

  • More stripe integration
  • making a fee calculatator- a lot of math and then had to use dinero.js for calculating money

Charge Types

Yesterday, I thought I was going to use Destination Charges. But now I realized I should use Separate charges and transfers. (Update: Not True- see next day's entry)

I played with the API and made test destination charges. But I didn't like them for WishTender. Destination charges put the entire payment in to the wishers account. Then subtract fees out of it. (Update: Not always true- destination charges can be set up differently. See next day's entry.)

While, I could make that work, I think it's messy and confusing to the wisher. I just want the price of the gift to go into the wishers account. In my market research, wishers didn't want to see the fees on their side.

Stripe Rounding Rules:

Stripe rounds the Stripe fee to the nearest unit (e.g., cents). For example, if the fee is 0.025, Stripe will round up to 0.03. If the fee is 0.024, Stripe will round down to 0.02.

Why don’t my numbers add up in Javascript?

So you’ve written some absurdly simple code, say for example:

0.1 + 0.2

and got a really unexpected result:

0.30000000000000004

Basic Answers:

Because internally, computers use a format (binary floating-point) that cannot accurately represent a number like 0.1, 0.2 or 0.3 at all.

When the code is compiled or interpreted, your “0.1” is already rounded to the nearest number in that format, which results in a small rounding error even before the calculation happens.

...Decimal numbers cannot accurately represent a number like 1/3, so you have to round to something like 0.33 - and you don’t expect 0.33 + 0.33 + 0.33 to add up to 1, either - do you?

Exact Types Floating-point cheat sheet for JavaScript

Precise Financial Calculation in JavaScript. What Are the Gotchas?

You should probably scale your decimal values by 100, and represent all the monetary values in whole cents. This is to avoid problems with floating-point logic and arithmetic. There is no decimal data type in JavaScript - the only numeric data type is floating-point. Therefore it is generally recommended to handle money as 2550 cents instead of 25.50 dollars.


Sunday, 10/18/20

Daily WishTender Update

Today's Progress:

  • More stripe integration docs
  • Tested the strip API in a new practice project

More Stripe notes

For a market place, I want to follow the Collect Then Transfer guide

I'm not sure about US tax reporting. I talked to the stripe chat support and they helped me a bit.

Bold means I'm using it:

Connected Accounts

  • Standard
  • Express
  • Custom

Specific Guides

  • Collect payments then pay out guide
    • For marketplaces where the money goas through the platform. Collect from user A. Payout to User B.
  • Enable your users to accept payments guide
    • Looks like this is also for marketplaces. But the payments go through the user instead of the platform?
  • Pay out money guide
    • Taking money from your platforms bank account, adding it to stripe, then payout connected accounts.

Charge Types

  • Direct charges

    • Customers directly transact with your user, often unaware of your platform's existence
    • A single user is involved in the transaction
  • Destination charges

    • Customers transact with your platform for products or services provided by your user
    • A single user is involved in the transaction
  • Separate charges and transfers

    Any one of these instances:

    • Multiple users are involved in the transaction
    • A specific user isn't known at the time of charge (like: which Lyft driver is coming?)
    • Transfer can't be made at the time of charge


Saturday, 10/17/20

Daily WishTender Update

Today's Progress:

  • Stripe integration docs

Stripe notes

Documentation

If you are just using stripe to get payments, it looks pretty easy. Academind shows you how to do this in their shopping cart Node.js videos. But there are less videos on connected accounts. So for those I had to go to the documentation.

The stripe documentation looked intimidating but it's actually really great. Don't be intimidated by it. Just sit down and start reading. It will start making sense if you give it some time. And take advantage of the 24/7 support if you need.

Connected Accounts

Connected accounts allow you to pay out customers, useful for marketplaces. Lyft uses stripe connected accounts to pay out their drivers.

For connected accounts there are 3 types. I want Express or Custom. Standard is cheaper but it is more difficult for the wisher to use. It would allow tenders to interact with wishers about disputes. I don't want that because crazed fans might make up disputes to try to interact with the wisher.

With standard the bank statement would have the wishers information. I want to protect the privacy of the wisher. I only want the platforms name on the bank statement- WishTender.

With express or custom, I can transfer funds. This makes WishTender the middle man.


Friday, 10/16/20

Daily WishTender Update

Today's Progress:

  • Fixed Tests: 133 passing
  • Cart Service and tests
  • Helper for seeding the database during test
  • Set up debugger for recursive tests

Total Tests Passed: 135


Thursday, 10/15/20

Daily WishTender Update

Today's Progress:

  • Cart Model/Test

Video on seeding mongo db

CSRF protection

Express validation

  1. Session store shopping cart
  2. Cart Model


Wednesday, 10/14/20

Daily WishTender Update

Today's Progress:

  • Order Model
  • Order Service
  • Order Tests


Tuesday, 10/13/20

Daily WishTender Update

Today's Progress:

  • Learning more about stripe
  • Started Order model

I can't wait to have actual payments in here!

Whiteboard

I couldn't wrap my head around the stripe integration. What should my next step be?

Thinking through drawing helped. I drew out the shopping flow. This helped me see my next move; make an order model.

I asked on twitter:

One of my fears as a new developer is that I’ll code something that works but isn’t scalable or optimal and fixing it later (especially after it’s in production) will be much harder than coding it perfectly today.

Is this fear warranted?

Answers

These answers really reminded me to stop trying to be so perfect. Too many times people have told me to move faster: Fail fast. I'm saving these as a reminder:

  1. Joey Doughty @ImprovementGeek:

    Perfect code is expensive.

    Refactors are part of the process.

    We just have to do our best and move on.

  2. Ant Pace @pacea87:

    I think it's better to get something working (minimal viable product) that people can use ASAP. Fail fast, learn fast.

    That being said, yes, technical debt is a real thing. But it's usually worth it, especially early on

  3. SAMY DINDANE @SamyDindane:

    At some point, you're going to get used to pushing code that isn't optimal, let alone perfect.

    The most important thing is to isolate that code + acknowledge that it's not optimal.

Code Resources

How To Use Stripe Connect to build market places

Build a market place in 5 min

Featured Teachers:


Monday, 10/12/20

Daily WishTender Update

I took off the weekend to work on a side project you can find here. It's an interactive javascript map that shows the effect of voting for a third party by state. I made this because people always say a 3rd party vote is a wasted vote. But I wasn't sure if that was true. I scraped the web and collected data into a google sheet using google sheets api. It was cool to edit google sheets using node Js. Then I used a library that turned the data into an interactive map.

Today's Progress:

  • Added a test for email (very involved!)
  • Started looking at stripe integration

Code Resources:

TESTING SMTP/Email

Mailtrap to test emails

Featured Teachers:


Friday, 10/9/20

Daily WishTender Update

Today's Progress:


if (!user.confirmed) {
logger.log('silly', `User account not confirmed.`);
return done(null, false, { message: `User account not confirmed.` });
}


Thursday, 10/8/20

Daily WishTender Update

Today's Progress:

  • made an Email, ThankYouEmail, and ConfirmationEmail class

  • made email Token model

  • figured out how to debug while running a mocha test- very useful. Next I got to learn how to run nodemon and mocha at the same time.

  • added Token model to user service and passed test

  • Created an HTML email with the team to send to beta users

    I just stole the HTML from another email and changed it a bit.

Tests Passed: 119

Now since I added the token, looks like some of my tests broke. I think if they were really unit tests the wouldn't break, but they're more like smaller integration tests.

Code resources:

TESTING SMTP/Email

flow to confirm email address

code to confirm email address


Wednesday/Thursday, 10/7/20

Daily WishTender Update

Today's Progress:

  • I moved today so limited progress, but I spent Wednesday to Thursday morning reviewing SOLID principles.

SOLID code principles help keep code maintainable and scalable. When I did smaller projects that wasn't important. But WishTender is a full production app. So I'm prioritizing it's maintainability.

Code Notes On SOLID:

It's not necessary or ideal to learn them in SOLID order. Here's the order of my notes:

  1. Dependency Inversion Principle
  2. Interface Segregation Principle
    • Composition vs Inheritance
  3. Liskov Substitution Principle
  4. Open–closed Principle
  5. Single Responsibility Principle

Dependency Inversion Principle

High-level modules (modules, functions, classes closer to view layer) should not depend on low-level modules (modules like a 3rd party API like Stripe). Both should depend on abstractions (e.g. interfaces).

No!:

STORE---->STRIPE

YES!:

STORE---->PAYMENT PROCESSOR<----STRIPE

const store = new Store(new StripePaymentProcessor("Dash"));
store.purchaseBike(2);

const store2 = new Store(new PayPalPaymentProcessor("Dash"));
store.purchaseBike(1);

This way if you change to paypal you just send in a different payment processor object- a PayPalPaymentProcessor. You don't have to change the (Store) to interact with paypal now, which would lead to way too much refactoring.

Video

Interface Segregation Principle

Interface means 'class' essentially. In other languages an interface is a class like object. But interfaces don't exist in javascript.

Break out your interfaces or large classes into smaller components or smaller interfaces so subclasses don't extend from classes that have methods that they don't use.

class Entity {
  constructor(name, attackDamage, health) {
    this.name =name
    this.attackDamage = attackDamage;
  }
  move(){...//bla bla bla}
  attack(){...//bla bla bla}
}

class Rock extends Entity {
  ...//bla bla bla
  move(){return null} //rocks don't move
}
class Character extends Entity {} //characters do move

Rocks don't move, but they inherit from Entity with has the move() method. Refactor this do use composition.

class Entity {
  constructor(name) {
    this.name = name;
  }
  // no more move() or attack()
}

const mover = {
  move() {
    console.log(`${this.name} moved`);
  },
};

class Character extends Entity {
  constructor(name, attackDamage, health) {
    super(name);
    this.attackDamage = attackDamage;
  }
}

Object.assign(Character.prototype, mover);

Now we can add the move() function to only classes that move.

Video

Composition vs Inheritance

When classes inherit you're describing 1. what objects are and 2. how they're related. But it doesn't always make sense.

class Monster {
...
}
class FlyingMonster extends Monster{
...
  fly(){...
...
}
class SwimmingMonster extends Monster {
...
  swim(){...
...
}
class SwimmingFlyingMonster extends FlyingMonster or SwimmingMonster?????{
...

SwimmingFlyingMonster inherit from SwimmingMonster or FlyingMonster?

With composition you're describing what an object can do.

function swimmer({ name }) {
  return {
    swim: () => console.log(`${name} swam`),
  };
}

function swimmingMonsterCreator(name) {
  const monster = { name: name };
  return {
    ...monster,
    ...swimmer(monster),
  };
}

Now you can compose a monster by adding the functionality it needs.

Video

Liskov Substitution Principle

if S is a subtype of T, then objects of type T may be replaced with objects of type S

In other words, if you have a function that excepts an instance class like Animal, every single subclass of that class (Dog, Turtle) must be able to able to enter that function and work properly.

function procreate(animal){...}

procreate(new Animal()) //...fine
procreate(new Dog()) //...fine
procreate(new NeuteredDog()) //...this wont work!

You might refactor this by making another subclass- FertileAnimal- and making the function-procreate accept an instance of that new subclass and it's subclasses:

class FertileAnimal extends Animal{...}
class Dog extends FertileAnimal{...}

function procreate(FertileAnimal){...}

procreate(new FertileAnimal())
procreate(new Dog())

Or you might change this from inheritance to composition.

const procreator = {
  procreate() {
    console.log(`${this.name} procreated`);
  },
};
class Dog extends Animal {
  constructor(name, barkSound) {
    super(name);
    this.barkSound = barkSound;
  }
}

Object.assign(Dog.prototype, procreator);

Video

Open–closed Principle

"software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification";

This means if you have an entity, meaning...like... a function, you shouldn't have to 'open' up and 'modify' the function to accommodate new circumstances. But you should be able to extend the function.. meaning...like... pass in new kinds of arguments.

Print Quiz Example

function printQuiz(questions){
  questions.forEach(question=>{

  if(question.type == 'multiply choice'){
    question.options.forEach(()=>{
      console.log(`${index+1}. ${option}`)
  }
  else if(question.type == 'shortAnswer'){
    console.log('Answer__________')
  }
  else if(question.type == 'shortAnswer'){
    console.log('Answer___')
  }
}

Switch statements or bunch of if statements are often a sign you are violating the open/closed principle.

Here we have a function that isn't closed for modification. Because if we have to accommodate a new type of question, let's say a true or false question, now we have to go in and add another else if statement.

function printQuiz(questions){

  ... // all the other code from above

  else if(question.type == 'trueOrFalse'){
    console.log('True or False')
  }
}

Refactor this by breaking out the types of questions into classes or functions.

class MultipleChoiceQuestion {
  constructor(description, option) {
    this.description = description;
    this.options = options;
  }
  printQuestionChoices() {
    this.options.forEach(() => {
      console.log(`${index + 1}. ${option}`);
    });
  }
}
class TrueOrFalseQuestion {...//more code}
class ShortAnswerQuestion {...//more code}

const questions = [
  new ShortAnswerQuestion("What does SOLID mean?"),
  new MultipleChoiceQuestion("Which SOLID principle is best?")
]

function printQuiz(questions) {
  questions.forEach((questions) => {
    console.log(questions.description);
    question.printQuestionChoices();
    console.log("");
  });
}

Now you can always extend printQuiz() by adding a new class of question without modifying the printQuiz() function itself.

The Open/Closed principle says you should add new code instead of modifying code. However, this isn't always true. Don't avoid modifying code all the time.

Video

Single Responsibility Principle

All of your classes, modules, function should have one single responsibility: One reason to be changed.

class CalorieTracker {
  construct(...//bla bla bla){
    ...//bla bla bla
  }
  trackCalories(...//bla bla bla){
    ...//bla bla bla
  }
  logCalories(...//bla bla bla){
    ...//bla bla bla
  }
}

This CalorieTracker class has two reasons to be changed; 1. If you need to change the way calories are tracked or 2. If you need to change the way you log calories.

Logging calories isn't directly related to tracking calories. So it's better to move it out.

function logCalories(someParameter){
    ...//bla bla bla
  }

class CalorieTracker {
  construct(...//bla bla bla){
    ...//bla bla bla
  }
  trackCalories(...//bla bla bla){
    ...//bla bla bla
    logCalories(this.blablabla){
  }
}

Now you only change CalorieTracker if you need to change how you track calories. If you need to change how you log calories you change logCalories.

Video

Featured Teachers

Tuesday, 10/6/20

Daily WishTender Update

Today's Progress:

  • Implemented nodemailer
  • reviewed SOLID

Code Resources:

NodeJs file Structure


Monday, 10/5/20

Daily WishTender Update

Today's Progress:

  • Updated WishlistModel to have owner
  • Added to wishlist routes
  • Updated WishlistItemModel to have owner
  • Added to wishlistItem routes
  • Finished addingItemToUser integration test

Total Tests Passed: 118


Sunday, 10/4/20

Daily WishTender Update

Today's Progress:

  • Added to wishlist routes
  • Added to integration test

Total Tests Passed: 107

Started a side project

Made a Google Sheets API example project. This sample project shows how to use Node.js to read and write to a google sheet.

This is for a project where I'll calculate the statistical effect of a singular vote on the 5% threshold vs. the presidential outcome.

What is the 5% threshold? If a third party gets 5% of the popular vote in the presidential election it becomes a major party. Meaning we would no longer have a two party system. Many people believe by increasing the number of parties, we increase competition among parties, which will encourage better ideas.

So my project will calculate if your vote has a greater effect on that %5 popular vote or a on the presidential outcome. This depends on your state. It will help you make an informed decision on how to use your vote.

Code Resources:

mongoose populate video


Saturday, 10/3/20

Daily WishTender Update

Today's Progress:

  • The team did market research and branch building on a forum.
  • Finished alias routes and integration tests for adding an alias to a user

Total Tests Passed: 106


Friday, 10/2/20

Daily WishTender Update

Today's Progress:

  • Learned about connecting the front and backend and reviewed some React
  • User routes and tests for delete end point
  • Integration test for creating a user and adding an alias
  • Alias routes: post, started put, authorization

Next Day Reflection:

I could have accomplished more. But I got distracted with a thread on Facebook. I'm embarrassed by that.

It was on a topic that I censor from my Facebook wall -politics. I was at fault because I actively searched out a political topic in the search bar. Subsequently, I found a thread started by a friend. In a sub thread of the post, an argument became uncivil. That stuff really bothers me. So I replied that this was the meanest thread I'd seen all morning. We then talked about about whether it's ok to be uncivil and insult people in discourse. It went on for a bit.

To end it, I "liked" the last comment even though I didn't agree with all of it. Then I stopped replying.

I learned this from 5harath who shared @JamesClear's advice:

“You’re probably right” is becoming one of my favorite phrases.

Whenever someone disagrees with you on a small matter (read: most things), you can just shrug, say “you’re probably right” and move on.

Not caring about winning trivial arguments saves so much time and energy.

Featured Teachers:

Code Resources:

How to implement server-side rendering in your React app in three simple steps


Thursday, 10/1/20

Daily WishTender Update

Today:

  • Wrote an authorization/access control middleware and added it to some of the user routes.
  • Now the debugger runs with nodemon.
  • Connected the debugger to the test database instead of the development database.
  • Added sessions to tests- retain cookies with chai-http.

Daily Tracking --> Motivation

Tracking my daily progress is pushing me to get more done. Sometimes, I'll get distracted by learning. Instead, today I moved fast doing. Don't get me wrong, learning is important. But it's easy to get caught up in excessive learning.

It's nice to see 4 accomplishments finished. It's motivating.

Yesterday:

  • I reviewed our survey for the "tenders" aka the gifters
  • The team sent out surveys for tenders
  • I learned about access control, JSON Web Tokens, and sessions

Express Security

Best Practices: Express Security

Secure headers with helmet.js


Wednesday, 9/30/20

September Progress

Last month I wrote where I was at the end of August.

Journal entry from 9/1/20:

9/1/20

Today WishTender has a front end with a home page, a menu that doesn’t have links, a profile page component, an add wish page component. None of it is linked to the backend except for a page that isn’t real that’s adds wishes and the scraper. The front end is not organized and isn’t SOLID. No tests.

The backend is more organized and has tests. There are wish services and you can scrape a product for info. There’s authentication but not social. Users have no connection to the wishes. No way to add credit card info. No logger. No templates.

We don’t know if we’re going with stripe or what. The survey said PayPal looks like a no go.

I have to add logging to the back end. Learn best practices for react, unit tests, and state management and update. It’s not SOlID

Then vs Today:

  • "There are wish services"- Today, I have models and services for users, wishlists, wishlist items, and aliases.

  • "We don’t know if we’re going with stripe"- We decided to start with stripe.

  • "I have to add logging to the back end"- I added winston logging.

  • I also have a around 90 tests and started adding to the user routes. Some are unit tests but some are more integration/functional.

  • I set up landing page and collecting beta user emails on mailerlite. We have 10 subrscibers! (well, 2 of them are me...)

  • My team launched social media and made a terms and conditions draft.

It doesn't seem like I got that much farther, even though I worked so much. I think I spent a lot of time learning about nodejs architecture and mongodb.

Tracking Progress

I want to move faster.

So I'm going to keep track of what I get done every day to keep me accountable:

Yesterday for WishTender:

  • I added a logout route.
  • I fixed my debugger which wasn’t working with dotenv
  • I watched the LinkedIn chapter of a tutorial on sessions
  • I looked at my sessions on robo3t
  • I posted a question and answer on stack Overflow
  • Started trying to understand how to authorize users to change their resources: their wishlists, their profiles, etc. while keeping them from updating others' resources.

How to only allow a user to update their own resources?

I was wondering how to prevent users from updating other user's resources. I thought this was called authorization, vs authentication where a user logs in. But it looks like it's called access control.

I found this question: Allow users to only update and delete their own object

Which led me to these two articles:

I also found a Guide to node.js authentication from Software On The Road, one of my favorite sites about Node.js architecture.

Featured Teachers


Tuesday, 9/29/20

Node.js VSCode Debugger Missing Env Variables

My debugger wasn't getting access to my .env environment variables. These hold my database uri's. So I would get this error when trying to run the debugger:

MongooseError: The `uri` parameter to `openUri()` must be a string, got "undefined". Make sure the first parameter to `mongoose.connect()` or `mongoose.createConnection()` is a string.

I fixed it by adding this to my launch.json file in my .vscode workspace folder. I added two values to the configurations object:

"env": { "PORT": "4000" },
"envFile": "${workspaceFolder}/backend/.env"

Saturday, 9/26/20

Silly Logging vs Verbose

silly logging is typically used to log EVERYTHING (Each function call, action, variable, extra information, whatever), whereas verbose logging is closer to debug logging, providing a logged pseudo-trace through functions and logs of certain critical variables.

-source: winston github


Wednesday, 9/23/20

Code next steps:

  • UserService
    • Email confirmation
    • change password
  • auth social login
  • routes

Let's get the routes in order today.

When to throw an exception?

Now that I'm moving into the routes I'm still thinking about when to throw exceptions, now as it pertains to the routes. Like when user logs in and the password is incorrect. Is that exceptional?

There were a lot of different nuanced opinions. This was the top answer.

My personal guideline is: an exception is thrown when a fundamental assumption of the current code block is found to be false.

Example 1: say I have a function which is supposed to examine an arbitrary class and return true if that class inherits from List<>. This function asks the question, "Is this object a descendant of List?" This function should never throw an exception, because there are no gray areas in its operation - every single class either does or does not inherit from List<>, so the answer is always "yes" or "no".

Example 2: say I have another function which examines a List<> and returns true if its length is more than 50, and false if the length is less. This function asks the question, "Does this list have more than 50 items?" But this question makes an assumption - it assumes that the object it is given is a list. If I hand it a NULL, then that assumption is false. In that case, if the function returns either true or false, then it is breaking its own rules. The function cannot return anything and claim that it answered the question correctly. So it doesn't return - it throws an exception.

This is comparable to the "loaded question" logical fallacy. Every function asks a question. If the input it is given makes that question a fallacy, then throw an exception. This line is harder to draw with functions that return void, but the bottom line is: if the function's assumptions about its inputs are violated, it should throw an exception instead of returning normally.

The other side of this equation is: if you find your functions throwing exceptions frequently, then you probably need to refine their assumptions.

Since logging is asks the question, do you have the credentials? I believe this makes sense to abstain from throwing an error when the password is wrong. Because you can answer that question, "No". However you might throw an error if null was passed in as the password.

Should it error? Put the function in the form of a question to find out.

But some functions don't ask clear questions. Like what about updateUser? This is a command.

For my updateUser, I return the updated user. I throw an error if the user can't be updated for some reason. What is the question here? Can you update user? or What does the user look like after we update it?

Well in my case, it seems like the question is the latter. Because we return the updated user. Now, if we can't update the user, we can't answer that question. So I must throw an error.

So I suppose to figure out the question your function asks, look at not just the name and usage of the function, but also what it returns.

Others were more exception-averse

I think you should only throw an exception when there's nothing you can do to get out of your current state. For example if you are allocating memory and there isn't any to allocate.

Because they're things that will happen normally. Exceptions are not control flow mechanisms.

But that doesn't seem to be a common principle. Many people disagreed in the comments. They thought exceptions are control flow mechanisms.

My intuition is that throwing exceptions to a central location will help with debugging. Especially, if you use a custom error. But it depends on what your question is asking.

With so many people disagreeing, it's almost like it doesn't really matter

Most important:

  • Be consistent with some principle for the way your app handles exceptions
  • Document it

Better Express Routing for Node.js

Route configs are responsible for initializing and associating routes with controllers.

Huh? What are route configs?

Controllers are responsible for taking the inputs from the route and invoking the appropriate actions to execute.

Huh? I though controllers were routes?

Ah I see:

Routes/Route Configs:

let router = express.Router();

router.get("/", homeController.index);
router.get("/info", homeController.info);

Controllers:

function index(req, res) {
  res.render("home/index", {
    title: "Home",
  });
}

function info(req, res) {
  res.render("home/info", {
    title: "More info",
  });
}

I lumped them together in my head.

Service layer vs microservice?

...a service can holds two purposes:

Operate as a micro-service - perform some CRUD (create, read, update, delete) against some data or database

Orchestrate several micro-services - combine and interact with different micro-services to perform some sort of business logic or complex situation

-source: caffeinecoding

Huh? I thought microservices and service layers had nothing to do with one another.

Each service has its own unique and well-defined role, runs in its own process, and communicates via HTTP APIs or messaging. Each microservice can be deployed, upgraded, scaled, and restarted independently of all the sibling services in the application.

-source: nodesource

So this^ is what I thought a microservice was: a mini app that "runs in its own process".

It looks like this caffeinecoding might be uses the term "mircro-service" to mean something else.

Featured Teachers


Tuesday, 9/22/20

Partnerships

Shlo, my team member and best friend, saw an up-and-coming OnlyFans competitor called Poststar.

Shlo suggested:

"Let’s contact them and offer a referral percentage if they promote our site!"

I thought: hmm a partnership is a great idea. But I know nothing about partnerships. However, my cousin has experience with partnerships. Let’s contact him.

Consultation about Partnerships

My cousin had a lot of experience with partnerships at his startup. He taught us a lot and pointed us in a better direction.

My Cousin's Advice On Partnerships

Good partnerships are hard to find, but when you do find them they are amazing for your business.

-something my cousin sort of said

1. Cast a wide net

WishTender shouldn't only focus on Poststars. Reach out to many companies you can partner with: OnlyFans and all their competitors.

2. Start Small

In addition to companies, reach out to potential users you can partner with. Partnering with users may mean getting them to be beta users or offering them a referral percentage for getting more users to sign on. If users pay for your software, it can mean giving them a free subscription.

These early users can help you make partnerships with big companies. It shows the big players that real people want this and use it.

As an example, my cousin said he and his team got beta users for their startup by first contacting people they knew. After some people they knew signed on, they went to everyone else and said, look so-and-so uses our product.

3. Empathize with their motivations

Send companies an email with a few bullet points on WishTender, how it can make them money, and a 30-60 second video demo. Not longer than 60 seconds, or they'll lose attention.

Empathize with their motivation:

When models share amazon wishlists, you're losing out on $1,000,000 extra a year you could make by having a white label wishlist.

That's pretty motivating.

Put in some work to figure out how much money what you might be able to bring them.

We can do a similar email to the users, as well.

4. Think Bigger

Instead of offering companies a percentage of what they make on their referrals to WishTender, offer them the ability to have a wishlist on their site using our tech. Charge them some amount to use it per month. Ex: 50k a month. Or combine some sort of referral with a charge.

I asked:

"How would we do that? We are a web app with our own site. Now you're saying to make a software for other companies?"

He said it could be done even with just an iframe. But whatever you decide, you can tell them; if they want it built, it will cost $X to make. Have them cover the expense of building it. You don't have to figure out how to build it first.

These companies have budgets put aside for stuff like this, and you can often charge more than you think. The idea isn't to take advantage of these budgets, but to give them something better for more. Think about big win/wins!

5. Cast a Net Early

Partnerships take long. You could contact a company today. But if they have a huge project going on, they may not be ready to partner with you for three months.

You don't need a finished product to contact companies about partnerships. Just a video demo. So contact them early, even if your product isn't ready.

6. Don't Fear Getting Your Idea Stolen

Don't be afraid that by emailing companies your idea, they will implement it behind your back. Good ideas are cheap. Execution is what counts. If these companies steal your idea, they were probably just going to implement it already.

This fear will stop you from being aggressive about partnerships. Cast your net early and wide. Don't let fear stop you.


My Startup - Tuesday, 9/22/20

Last update, I revealed my startup: WishTender, a wishlist site for public personalities.

This week I continued to work on the backend. I learned a lot about error handling. And we started social media!

WishTender Timeline

I was wondering how long I'd been working on WishTender. Here are some dates:

  • 6-21-20: My sister and I start talking about the idea
  • 7-3-20: I made a folder called "gift_registry_business_idea"
  • 7-8-20: Started making a demo based off of blueprint registry
  • 7-10-20: Showed the demo to my sister and spouse. Which got them to want to participate.
  • 7-13-20: Started building the react frontend.
  • 7-20-20: Bought Wishtender.com domain.
  • 7-21-20: Started building the node.js backend
  • 7-31-20: Started sending out market research surveys
  • 9-15-20: Deployed a landing page to collect beta users
  • 9-16-20: Started promoting on social media

Meeting with a Lawyer

This week a lawyer friend gave us some advice on protecting ourselves as an LLC and creating a terms and agreements. She suggested getting inspiration from different site's terms and conditions that are similar to WishTender.

Resources warn that if you copy the terms and conditions too closely, it's copyright infringement. But my entrepreneur neighbor, Moishe, said not to worry until after you get a cease and assist.

Telemeeting with lawyer and rebbetzin Laura E. in the conference bed

Social Media

We started doing social media last week. 52 followers on twitter. We tweeted and pinned my email up so people can ask me questions and make suggestions. One potential user responded. She's been emailing lots of great input. She was one of the people we surveyed last month.

Code Notes And Non-Code

There are two more non-code related sections in this update:

They are mixed in with code notes below.

Error Handling

Why do we handle errors? Why not just fix the actual problem? I'm confused.

So if something goes wrong in my app, I have this error handler

// error handling function
app.use((err, req, res, next) => {
  if (res.headersSent) {
    return next(err);
  }
  console.err(err);
  return res.status(500).render("500", {
    title: "500",
  });
});

But I'm confused, why should I be planning for errors to happen. Why not just fix the code before they happen? I think this has to do with errors vs bugs.

Operational errors vs. programmer errors:

People use the term “errors” to talk about both operational and programmer errors, but they’re really quite different. Operational errors are error conditions that all correct programs must deal with, and as long as they’re dealt with, they don’t necessarily indicate a bug or even a serious problem.....

By contrast, programmer errors are bugs. They’re cases where you made a mistake, maybe by forgetting to validate user input, mistyping a variable name, or something like that. By definition there’s no way to handle those. If there were, you would have just used the error handling code in place of the code that caused the error!

This distinction is very important: operational errors are part of the normal operation of a program. Programmer errors are bugs.

Operational Errors:

  • failed to connect to server
  • failed to resolve hostname
  • invalid user input
  • request timeout
  • server returned a 500 response
  • socket hang-up
  • system is out of memory
  • out of memory or too many open files
  • system’s configuration (e.g., no route to a remote host)
  • remote service (e.g., a 500 error, failure to connect)

Programming Errors aka Bugs:

  • forgetting to validate user input
  • mistyping a variable name
  • tried to read property of “undefined”
  • called an asynchronous function without a callback
  • passed a “string” where an object was expected
  • passed an object where an IP address string was expected

Failure to handle an operational error is itself a programmer error.

How do you change a database later?

I'm super nervous about making any decisions with the database. Because what happens if we want to change something? Like a add a new type of user?

Database Schema Migration

I found out these are called migrations.

Database Schema Migration is the process of updating a Collection Schema and/or adapting the existing data for new requirements.

What if I start with one type of user that has name, password, and email. Regular user fields. But because all users are wishlist creators, the user schema also has wishlists. What happens 2 months from MVP, when I want to add users who are gifters? They would have name, password, and email but they wouldn't need wishlist. And they may need another field that wishers wouldn't need. Like giftsPurchased. I think I would want to change the schema. But could that be an issue?

I guess migrations can help with this.

Error Handling Again

I log EVERYTHING but not everything is printed in the output log 😉

I strongly believe that you need to log when an action is about to be performed, when the action has been performed, the result and the error if happened.

-source: softwareontheroad

Log:

  • when as action is about to be performed
  • when the action has been performed
  • the result and the error if happened

What exactly is an action? What is one unit of action? Maybe like an api call.

forward your errors all to the same central location in your server.

Don’t obfuscate your errors, be honest, let your users know why the request fail, so they can perform another action, or try something different.

A good error message will be like:

The user search engine doesn’t work for now but you can still view your profile.

-source: smashingmagazine

For example, a JavaScript developer may decide to throw in a number instead of an error object instance, like so:

// bad
throw "Whoops :)";

// good
throw new Error("Whoops :)");

You might not see the problem in throwing other data types, but doing so will result in a harder time debugging because you won’t get a stack trace and other properties that the Error object exposes which are needed for debugging.

-source: smashingmagazine

TRANSFORMING ERROR TO STRING- Dont!

reject("Template not found. Error: ", +err);

This approach has a lot of downsides because when the concatenation was done, the developer implicitly runs toString on the error object returned. This way he loses any extra information returned by the error(say goodbye to stack trace). So what the developer has right now is just a string that is not useful when debugging.

A better way is to keep the error as it is or wrap it in another error that you’ve created and attached the thrown error from the databaseGet call as a property to it.

-source: smashingmagazine

Errors are thrown by the engine, and exceptions are thrown by the developer.

-source: rollbar

The important here is not to handle the errors from the underlying layers but to throw it to the controller layer.

-source: softwareontheroad

import Logger from "../logger";
import UserService from "../services/user";

export default (app) => {
  app.get("/user/search-location", (req, res, next) => {
    try {
      const { lat, lng } = req.query;
      Logger.silly("Invoking user service to search by location");
      const users = UserService.SearchUserByLocation(lat, lng);
      return res.json(users).status(200);
    } catch (e) {
      Logger.warn("We fail!");
      return next(e);
    }
  });
};

The controller layer just passes it to the next express >middleware, our centralized error handler.

import Logger from "../logger";
export default (err, req, res, next) => {
  Logger.error("Error %o", err);
  return res.json(err).status(err.httpStatusCode || 500);
};

-source: softwareontheroad

How do we know what next passes the error too?

I'm assuming that this centralized error handler is used in an app.use() call in the server file?

OK, so I think I have a better picture now of how I need to handle these errors!

In this pattern, we would start our application with an ApplicationError class this way we know all errors in our applications that we explicitly throw are going to inherit from it. So we would start off with the following error classes:

-source: smashingmagazine

That's smart, so if you see an applicationError, you know it came from an error you coded and not from some other middleware... middleware? Sometimes I say words and I don't know if they're the right ones. I'm not really sure what else throws errors. I think mongoose throws errors for example when there's problems with database queries. And is mongoose a middleware? Oh if we pass req or res to it then yes. But...we don't pass req or res to it, but we do pass information we got from req or res to it. Hmm so I think it's not a middleware.

I think I can't... I think I can

Yesterday, I felt so lost with all this error handling stuff. I was like, "what am I getting myself into?! I can't code this freaking app! I know nothing! Only a seasoned developer could build this app!"

24 hours later and I feel like I can code the next step.

Let that be a lesson to me: When I'm looking out into a very intimidating pile of information, I've always been able to find my next steps. This feeling happens over and over again:

  1. Oh no, I can't do it!
  2. Study the subject 24 hours
  3. Oh wait, I know enough to take a step!

No reason to catastrophize the situation when I haven't even put in a good 24 hours!

Even though I still experience this feeling, it has lessened in intensity. Overtime, I've learned to expect this pattern. So I don't freak out as much in the face of the unknown. However, with time I challenge myself more and more, so the unknown is more intimidating and challenging.

Social

It is so exciting to see people interacting with the social media account!

More Error Handling

I copied this from a tutorial.

module.exports = (err, req, res, next) => {
  if (res.headersSent) {
    return next(err);
  }
  logger.log("Error:", err);
  return res.status(500).render("500", {
    title: "500",
  });
};

Where does next(err) go when this is the last middleware in my app.js file?

If you pass an error to next() and you do not handle it in a custom error handler, it will be handled by the built-in error handler; the error will be written to the client with the stack trace. The stack trace is not included in the production environment.

-source: expressjs

This is the default error handler in express. That page writes in more detail about why you need if (res.headersSent) {return next(err);} in your error handler.

When to throw an exception?

I posted this on stackoverflow:

I have an updateDocument method in a class for a service layer in a node.js, express, mongoose application. I'm wondering what is the best practice for handing cases where the update method didn't update a document, for example if the wrong id is passed in.

Version 1: If the update wasn't successful, return an object with success: false:

  async updateDocument(id, updates) {
    const output = await this.DocumentModel.updateOne({ _id: id }, updates);

    let message = 'Something went wrong';
    let success = false;
    let updatedItem = null;
    if (output.nModified) {
      message = 'Successfully updated document.';
      success = true;
      updatedItem = await this.getDocument(id);
    }

    return { message, success, updatedItem};
  }

Version 2: If the update wasn't successful, throw an error:

  async updateDocument(id, updates) {
    const output = await this.DocumentModel.updateOne({ _id: id }, updates);

    let updatedItem;

    if (output.nModified) {
      updatedItem = await this.getDocument(id);
    } else{
      throw new Error("The document wasn't updated")
    }

    return updatedItem;
  }

Do you always throw an exception when the input, such as a bad id, isn't correct? Or could you return information about the update being a success or not? As newbie node.js developer, I'm not sure I am grasping the full picture enough to recognize problems with either method.

I asked this on stackoverflow.

When to throw an exception?

Every function asks a question. If the input it is given makes that question a fallacy, then throw an exception. This line is harder to draw with functions that return void, but the bottom line is: if the function's assumptions about its inputs are violated, it should throw an exception instead of returning normally.

Should a retrieval method return 'null' or throw an exception when it can't produce the return value?

Answer 1:

Whatever you do, make sure you document it. I think this point is more important than exactly which approach is "best".

Answer 2:

If you are always expecting to find a value then throw the exception if it is missing. The exception would mean that there was a problem.

If the value can be missing or present and both are valid for the application logic then return a null.

More important: What do you do other places in the code? Consistency is important.

Where should exceptions be handled?

Answer 1: in the layer of code that can actually do something about the error

Exceptions should be handled in the layer of code that can actually do something about the error.

The "log and rethrow" pattern is often considered an antipattern (for exactly the reason you mentioned, leads to a lot of duplicate code and doesn't really help you do anything practical.)

The point of an exception is that it is "not expected". If the layer of code you are working in can't do something reasonable to continue successful processing when the error happens, just let it bubble up.

If the layer of code you are working in can do something to continue when the error happens, that is the spot to handle the error. (And returning a "failed" http response code counts as a way to "continue processing". You are saving the program from crashing.)

-source: softwareengineering.stackexchange

Answer 2: Handle errors centrally, not within a middleware

Without one dedicated object for error handling, greater are the chances of important errors hiding under the radar due to improper handling. The error handler object is responsible for making the error visible, for example by writing to a well-formatted logger, sending events to some monitoring product like Sentry, Rollbar, or Raygun. Most web frameworks, like Express, provide an error handling middleware mechanism. A typical error handling flow might be: Some module throws an error -> API router catches the error -> it propagates the error to the middleware (e.g. Express, KOA) who is responsible for catching errors -> a centralized error handler is called -> the middleware is being told whether this error is an untrusted error (not operational) so it can restart the app gracefully. Note that it’s a common, yet wrong, practice to handle errors within Express middleware – doing so will not cover errors that are thrown in non-web interfaces.

-source; Handle errors centrally, not within a middleware

Answer 3: In the controllers

You can finally handle your errors in your controller classes.

-an answer to my stack overflow question

So it seems like these #1 & #3 disagree with #2. #1 says to handle it right away if you can. So for me it would be in the service layer or in the controller. But the #2 says handle it centrally, like in the server file. I went with #2.

My decision: throw the error in a custom error class

I combined a few methods people suggested. I am throwing the error, but I'm not "log and rethrow"-ing, as the answer above warned against. Instead, I put the error in a custom error with more information and throw that. It is logged and handled centrally.

So first in my service layer this is how an error is thrown:

async addUser(user) {
   let newUser;
   try {
      newUser = await this.UserModel.create(user);
   } catch (err) {
      throw new ApplicationError( // custom error
         {
            user, // params that are useful
            err, //original error
         },
        `Unable to create user: ${err.name}: ${err.message}` // error message
      );
   }
   return newUser;
}

ApplicationError is a custom error class that takes an info object and a message. You could put other helpful information in your custom error class, even maybe what EJS template to use! So you could really handle the error creatively depending on how you structure your custom error class. I don't know if that's "normal", maybe it's not SOLID to include the EJS template, but I think it's an interesting concept to explore. You could think about other ways that might be more SOLID to dynamically react to errors.

This is the handleError file for now, but I will probably change it up to work with the custom error to create a more informative page. :

const logger = require("./logger");

module.exports = (err, req, res, next) => {
  if (res.headersSent) {
    return next(err);
  }

  logger.log("Error:", err);
  return res.status(500).render("500", {
    title: "500",
  });
};

Then I add that function to my server file as the last middleware:

app.use(handleError);

In conclusion, it seems like there's a bit of disagreement on how to handle errors though it seems more people think you should throw the error and probably handle it centrally. Find a way that works for you, be consistent, and document it.

Loaders?

What is a loader?

Loaders are transformations that are applied on a resource file of your app. They are functions (running in node. js) that take the source of a resource file as the parameter and return the new source.

source

Sounds unrelated to what I need to know right now.

I Ran Into My CBD Entrepreneur Neighbor, Moishe

Moishe said not to be afraid of failure. He said fail fast. It's better than failing slow. Go fast, and don't be afraid to fail. But also don't make terrible decisions! Like Elon Musk says, there are type 1 and type 2 decisions: decisions that are consequential and irreversible and decisions that can be made fast and are reversible.

I posted another question: How to deal with Parent and Child mongoose schemas that have cyclic dependencies?

Featured Teachers

My Startup - Wednesday, 9/16/20

Up until this point, I've been updating you on my new startup in vague terms. But I've kept hidden the exact idea.

This week, we started posting on social. So I can finally talk about the startup.

Here it is:

WishTender

WishTender is a safe and flexible wishlist app for public personalities.

Currently, there are gift registry apps for the average person. However, for public figures with doting fans, these wishlists miss the mark.

Most wishlist apps reveal the wisher's address- Not something an internet celebrity wants their fans to know! Even if the wishlist hides the address, there are workarounds to find the address.

We've done a lot of market research with OnlyFans models. If you don't know what OnlyFans is, it's like patreon but too sexy for work. (Is TSFW the new NSFW? )

Fans want to buy them gifts. But the models are underwhelmed with the current wishlists- unsafe wishlists, slow orders, lack of gift options.

We're addressing it all and making the premier wishlist app.

What I'm up to with the business


Social Media

I have a Gen Z-er working on the social media content. She created some content that had me laughing. I want our brand to be really light and fun. After all, it's gifts!

Most of our customers are Gen Z and Millennial ladies. Mean Girls is my favorite movie, and I loved this reference. Nice job Gen Z content creator!

Landing Page/Email Signup

I now have a landing page with a beta user sign up.

I used:

  • Design:

    • Tech: React
    • Photos/Icons: Freepik $11.99
    • Logo: BrandCrowd $15
    • Adobe Creative Cloud $56.30 (was already paying)
  • Hosting: Netlify free

  • Business Email: Zoho free

  • Email Marketing: MailerLite free

    • For the beta signup I used Cuppa's beta signup as a model. I liked what KP did:

      WishTender:

BackEnd Coding

I continued to code the backend in Node.js, mongoose, and express. I'm writing a lot of tests, which I never really did in the past.

Build In Public?

Would you like me to share my journey building WishTender?

If so what topics are most valuable?

  • Coding?
  • Marketing?
  • What do people want to know from me?

And what medium would people like?

  • Twitter threads?
  • This Blog?
  • Youtube videos?
  • etc

I'd love to hear your thoughts in the comments on Twitter or LinkedIn. Or feel free to dm me.


Messy Coding Notes/Resources

The Data Access Layer- What is it in Node.js?

I kept reading about the data access layer (here and here) but didn't know what that meant. In one article it was called a "MongooseService" which took a model as a parameter. I didn't understand the point of that. It seemed like a pointless shell around a model that does all the same things the model does. Later I found this:

Models are the Data Access Layer

source

Hmmm... so the model itself is the data access layer. I don't know what that "MongooseService" was about...

The service layer should not "Directly interact with the database"(src) but what does that mean? Is using a model the same as directly interacting with the database? Or is the model an intermediary? I don't know.

I think the model is the intermediary, but I'm not sure. I'm just going to move on and assume it's fine to use the Model in the service layers since I see that all the time like here:

import UserModel from '../models/user';
import CompanyModel from '../models/company';

 export default class UserService {

  async Signup(user) {
     const userRecord = await UserModel.create(user);
...

Featured Teachers


Startup Update - 9/8/20

9/8/20

Business

We plan on launching beta in October. We're going to start doing social media this month. Once we do, I'll start talking about the nature of the startup, which I've kept secret until now.

Code

Did a lot of database work and set up logging with the winston module.

Can I Finish Beta By October?

I honestly don't know if I'll be able to finish by October.

  • I have never built a complete app
  • Less than 2 years ago I started 100DaysOfCode as a newbie.
  • I'm building this whole thing myself!
  • I'm learning as I build this.

But I'm going to do my best attempt.

Messy Notes On


  • logging
  • MongoDb relationships
  • MongoDb embedded documents

The test file for my web scraper logs the data for each product scraped. That means the price, product name, currency, and images for a dress from amazon, a pot from etsy, a supplement from Kion, headphones from eBay, etc..

Those logs help me remember which products return an empty value- something I might want to address later. But it's not very important. The scraper isn't necessarily for the app to work, it just increases usability.

So having all those logs is distracting. Yet I don't want to delete them, since rewriting them in the future when I want them would be a pain.

How do I manage this? I remember some article talking about loggers. Would a logger be some program that helps you log more or less information?

You might want to turn off logging if you're in a development environment vs a production environment. Or even if you're just testing locally on your machine or VM, if you've got a ton of logging for debug purposes or otherwise, that can really clutter up your console and you might want to just test with logging disabled for a bit...

...Popular Node logging frameworks like Winston and Bunyan allow for log levels, easy toggling logs on and off based on environment, and sometimes (in the case of Winston) support for custom log levels that you as a developer can define.

-Source: Should you use a logging framework or console.log() in Node?

LinkedIn Learning Why and what should I log?

Operational errors vs. programmer errors:

People use the term “errors” to talk about both operational and programmer errors, but they’re really quite different. Operational errors are error conditions that all correct programs must deal with, and as long as they’re dealt with, they don’t necessarily indicate a bug or even a serious problem.....

By contrast, programmer errors are bugs. They’re cases where you made a mistake, maybe by forgetting to validate user input, mistyping a variable name, or something like that. By definition there’s no way to handle those. If there were, you would have just used the error handling code in place of the code that caused the error!

This distinction is very important: operational errors are part of the normal operation of a program. Programmer errors are bugs.

pm2 starts node app correctly , but I am not able to access the app in browser:

To run npm start with pm2:

$ pm2 start npm -- start

MongoDB One-to-Many Relationship tutorial with Mongoose examples

nodejs-mongoose-one-to-many-relationship:

const createTutorial = function (tutorial) {
  return db.Tutorial.create(tutorial).then((docTutorial) => {
    console.log("\n>> Created Tutorial:\n", docTutorial);
    return docTutorial;
  });
};

const createImage = function (tutorialId, image) {
  return db.Image.create(image).then((docImage) => {
    console.log("\n>> Created Image:\n", docImage);
    return db.Tutorial.findByIdAndUpdate(
      tutorialId,
      {
        $push: {
          images: {
            _id: docImage._id,
            url: docImage.url,
            caption: docImage.caption,
          },
        },
      },
      { new: true, useFindAndModify: false }
    );
  });
};

Embedded sub documents vs Document references :

Your usage implies an embedded sub document inside the model which only requires a schema definition for the sub document. This will store both schema's in a single document in a single collection in MongoDB... If you want to keep the two models then you would need to use a reference in your Post schema instead.

Unique Deeply Embedded Mongoose Documents

Add a deeply embedded document conditionally if unique field value

Featured Teachers:

Not featured in this log, but helped me this week:

Startup Update - 8/29/20

8/29/20

I continued working on my startup.

This past week (or so) my spouse and I:

  • Business:

    • Spoke with accountants for tax advice
    • Spoke with a payment processing company
    • Researched payment processing
    • continued market research by sending out surveys on Twitter DM
    • Opened a business bank account

Our LLC was approved on Monday! This will help us proceed with some of our business tasks.

  • Code:

    • implemented unit tests
    • cleaned up a bunch of my code
    • learned about SOLID principles and node architecture
    • Worked on part of the app that scrapes online stores for product info. I used Test Driven Development for the first time.

Testing the web scraper:

Learning vs Doing


From the beginning, I've been telling myself to build. Don't worry about the "right way" to code. But as the code base gets bigger, it necessitates more organization. Otherwise, the messy, large code base becomes unmanageable.

To help; I've been implementing unit tests. Thinking about good code architecture. Learning about SOLID principles. AKA doing things the "right" way.

Coding with some structure, really helps my brain from exploding.

What's the Correct Balance?

However, I struggle with knowing the right amount of the "right" way. (Oh wow, that's like "right"-ception.) Basically, I can try to learn the correct way to code. But, there's always more to learn. More to perfect. Thus, this prevents me from doing. I get sucked into a learning purgatory. So when is learning about the "right" way no longer beneficial?

On the other hand, if I just code without learning the right way, I run into issues later.

I'm playing it intuitively for now. I will go on streaks of doing. Then streaks of learning. Each one feeds the next. When I coded something incorrectly it gives me context. So when I go off and learn the right way, I understand the consequences from experience.

Principles

When to learn and when to do, unfinished principles:

  • Play it by ear. Go with the flow.
  • If you have enough information to make, make.
  • If it's clear the way you're making something is causing issues, go learn the right way.
  • When you're making and you understand the context of what you are doing and continuing the wrong way will cause issues, go learn the right way. You've got enough context.
  • When you've learned enough to correct your mistake, make.
  • When you've learned for a while, if you're trying to get more details but it becomes increasingly difficult to find solid answers or there isn't an agreed upon "right" way- make! Pick one way or make something up. Later, you will have more context that will lead you to the "right" answer.
  • Let your doing guide your learning needs. Don't stick to someone elses recommended learning path. They aren't building what you're building.

I just brain stormed these. What do you think are some good principles for balancing learning with doing?? Or how would you better articulate what I'm trying to say? Let me know @DashBarkHuss

Revealing The Business


Last week I spoke about how I'm unsure when I should reveal the business. Thank you to those who reached out to me and to give me your thoughts. I still, am not sure. (As I said, I will tell anyone the idea in DM)

As I said before, what's my business moat? What's to keep someone with more money from seeing my market research and copying my idea?

But as someone who loves the idea of building in public, I really really want to share everything. I think it helps others. And I think it benefits me and my business. I'd be building reputations, trust, and networking.

I think once I get closer to a working product I will start talking about it. Then my moat is a little bigger. But I'm still curious what people think about this. Share with me @DashBarkHuss

Code Notes & Resources

Resources:

3-Layer Architecture

  1. Controllers receive incoming client requests, and they leverage services
  2. Services contain all business logic, and can also make calls to the data access layer
  3. The data access layer interacts with the database by performing queries
  4. Results are passed back up to the service layer.
  5. The service layer can then hand everything back to the controller
  6. 6.The controller can then respond to the client!
// services/PostService.js

const MongooseService = require("./MongooseService"); // Data Access Layer
const PostModel = require("../models/post"); // Database Model

class PostService {
  /**
   * @description Create an instance of PostService
   */
  constructor() {
    // Create instance of Data Access layer using our desired model
    this.MongooseServiceInstance = new MongooseService(PostModel);
  }
  /**
   * @description Attempt to create a post with the provided object
   * @param postToCreate {object} Object containing all required fields to
   * create post
   * @returns {Promise<{success: boolean, error: *}|{success: boolean, body: *}>}
   */
  async create(postToCreate) {
    try {
      const result = await this.MongooseServiceInstance.create(postToCreate);
      return { success: true, body: result };
    } catch (err) {
      return { success: false, error: err };
    }
  }
}

module.exports = PostService;

Bare Express

Featured Teachers:

Startup Update

8/25/20

I'm reading about SOLID code principles and good code architecture.

Routes vs Services

When coding the backend on Node and Express, I wanted to know what should be in the services code vs what should be in the routes code.

"Business logic" should be in services- but what constitutes business logic?

Do mongoose calls to the database constitute business logic? Or should those be in the routes?

This tutorial made it more clear how code should be separated among routes and services: Node.js Service Layer vs Routes vs Controller

SOLID

This article brought me to the concept of SOLID code principles.

Steven Lott recommends learning in this order instead of mnemonically (ILODS vs SOLID):

  1. Interface Segregation Principle
  2. Liskov Substitution Principle
  3. Open/Closed Principle
  4. Dependency Inversion Principle
  5. Single Responsibility Principle

I didn't watch all of these because I found it at the end of my research but heres a video series on SOLID.

Interface segregation principle

I kept seeing the word "client." But not how I'm used to the word being used- the client vs the server. What did they mean when they said client?

A client is something that is asking/requesting something from another class.

-Source: Interface segregation principle

By the way, this-> Interface segregation principle- was the best explananation of Interface segregation principle. Thanks @allanpweb. Interface segregation principle is one of the SOLID coding principles.

Still not sure what a client is in this context. It's a class that uses information from another class? Oh- is it just any part of a code that uses a class? Or is it an object that uses another class? What were the clients in the example they gave?

No client should be forced to depend on methods it does not use.

I think electricCar and motorCar were the clients of the Car in this example. They were forced to depend on methods they both didn't need.

electricCar didn't need

  • combustionEngine
  • fuelTank

motorCar didn't need

  • rechargableBattery
  • electricMotor
  • kilowatts

So these two instances were the clients of Car.

Ok got it.

The Liskov Substitution Principle: Subtypes must be substitutable for their base types.

Objects created from our FastVehicle aren’t completely substitutable for objects created from our base Vehicle constructor. Our FastVehicle violates the Liskov Substitution Principle!

-Source: SOLID JavaScript: The Liskov Substitution Principle

Mitigating LSP Violations

  • Contracts
    • Executable specifications
      • The contract for how a particular library is intended to be used is contained in a suite of automated tests.
    • Error checking
      • Error checking directly in the code itself in the form of preconditions, post conditions and invariant checks. Known as Design By Contract.

Always use Test-Driven Development to guide the design of your own code

Optionally use Design By Contract techniques when designing reusable libraries

  • Favor object composition over class inheritance
    • Composition
    • Inheritance

Video on Composition vs Ineritance

Inheritance vs. Composition in JavaScript

The polymorphism is a core concept of an object-oriented paradigm that provides a way to perform a single action in different forms. It provides an ability to call the same method on different JavaScript objects. As JavaScript is not a type-safe language, we can pass any type of data members with the methods.

-Source: JavaScript Polymorphism

Open/Closed Principle:

Open-Closed Principle means our JavaScript modules should be open to extension, but closed to modification.

Meaning that if someone wants to extend our module’s behavior, they won’t need to modify existing code if they don’t want to.

There’s a very easy rule of thumb you can follow here. If I have to open the JS file your module and make a modification in order to extend it, you’ve failed the open closed principle.

-Source: The 5 SOLID JavaScript

Dependency Inversion Principle

  1. High-level modules should not depend on low-level modules. Both should depend on abstractions (e.g. interfaces).
  2. Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.

Good example in this video: Video Dependency Inversion Principle

Single Responsibility Principle Good video on Single Responsibility Principle

Startup Update

8/21/20

The last few days I had family visit. I let myself enjoy time with family. Family is a priority. I found time to code throughout the day. But my focus was being present with family. It was the right thing to do, for me and my family.

Market Research


We redid our survey. We took the questions we were going to ask in the phone call follow ups, and put them in the survey instead. Many people didn't want to talk on the phone.

The survey focuses on two things:

  1. Customer satisfaction with their current product.
  2. Presenting our unique solution and seeing if they would like it.

We continued to cold Twitter DM the survey to people who fit our customer profile. Our response rate is above average. At least according to my mom! But for real, my MIL says that 8% is the average response rate. I think we're right around there.

I Haven't Said What The Business Is


Not publicly- but if you DM me I might tell you.

I really want to reveal what the startup is, publicly on my blog here. I think it would be helpful to readers.

But at the same time, there are benefits to keeping it quite.

  1. I don't want to get a false sense of accomplishment from talking about my endeavor before it's done.
  2. Business defensibility

Business Defensibility / Competitive Moat:

My friend Joseph Prosnitz told me about this idea.

The term “competitive moat” (popularized by Warren Buffett as an “economic moat”), refers to a business' ability to maintain competitive advantages in order to protect its long-term profits and market share from competing firms.

-Kristin Luck

So when Mr. Wonderful on Shark Tank asks:

"What's going to stop some other company from copying your idea and putting you out of business?"

Mr. Wonderful is asking: What is your business defensibility?

Business defensibility can come from patents or a good brand for example.

I believe our business defensibility will be our brand (TBD). But right now we have no brand. So I worry about talking about the idea publicly with no business defensibility.

I'm curious what you guys and gals think. Would you like for me to finally talk about the business idea on here? And do you think my concern is warranted?

No-Code

No-code is starting to peak my interest. No-code means building code without writing code. No-code solutions use graphical interfaces instead.

I wonder if it's really powerful and flexible, or if it would just slow me down. I will continue to code my app but I think this is an interesting topic. If no code, really works, it would be great for rapid prototyping.

Code Notes:


As my code gets bigger I keep needing to organize it more.

I started adding testing units.

I added ESLint. Here's some resources I used for Eslint.

More notes on Eslint. It's always been a pain for me to add!


To get rid of 'File is a CommonJS module; it may be converted to an ES6 module' warning:

CONTROL SHIFT P -> Open Settings (JSON)

Add:

  "javascript.suggestionActions.enabled": false,
  "typescript.suggestionActions.enabled": false

Keep getting this error: "ESLint: Failed to load config "airbnb" to extend from."

Tried this ESLint not working in VS CODE?:

CONTROL SHIFT P -> Open Settings (JSON)

Add:

"eslint.validate":[
   {
       "language":"vue",
       "autoFix":true
   },
   {
       "language":"html",
       "autoFix":true
   },
   {
       "language":"javascript",
       "autoFix":true
   },
   {
       "language":"typescript",
       "autoFix":true
   }
]

Restart VSCode and enjoy.

I don't think this worked ^

I moved the eslint stuff to the folders in backend, but I don't think it made a difference.

**FIXED!**

I finally fixed the error: "ESLint: Failed to load config "airbnb" to extend from."

I didn't have eslint-config-airbnb, I had eslint-config-airbnb-base. So I change my eslintrc.json package to have "extends": ["airbnb-base", "prettier"] instead of "extends": ["airbnb", "prettier"]

Startup Update

8-10-20


I have been working on my startup, which I'm calling Project WT. It's still a secret. A clue for context- it's sort of like a marketplace.

Code

I'm coding the app in Mongo, Express, React, and Node. This week I brushed up on Express skills and cleaned up some of my messy Express code. As my code gets bigger, it becomes more important to keep it organized.

I watch some of these two LinkedIn Learning tutorials:

Marketing

We (my spouse, sister, and I) did some market research in the form of a survey last week. This week we are doing follow ups by phone. The goal was to find out if our potential customers thought our solution would be helpful, and find out what blind spots we are missing.

I really struggled with the mock call we did. I'm supposed to lead the calls since I know the most about the project. My sister and spouse gave me a lot of feedback during the mock call. It was just way too much feedback.

I couldn't juggle all of their advice- be super friendly so they leave thinking "what a nice person!", don't say it that way, do it this way, have a script but don't sound like your reading from it etc. It was extra hard since it was a mock call and it was like acting- not real!

It all made me so nervous for the real calls.

So to ease my nerves, I decided they should not be on the call with me- I originally planned they would be silent but listening in. After the stressful mock call, this would make me way too self conscious.

And the second thing I did was I decided to discount a lot of their feedback. Too many pointers, and I'm going to freeze trying to achieve everything; And ultimately achieve nothing.

The Real Call

I was still pretty nervous. But when I did the first real call, of course it was just fine! No, I didn't woo them with my charisma- but I also didn't insult them either. I got the information I needed. It was polite. And everyone left with their day not any worse.

How I Managed My Feedback Overwhelm

So while the feedback had a negative effect on me, I still think it's always important for your team to give you feedback.

It's NOT your teams' job to censor their feedback. Instead, it's your responsibility to manage yourself when that feed back is overwhelming.

The ways I managed my feedback-overwhelm were:

  • Disregard the least important feedback
  • Recognized that not all the feedback you got is correct or important
  • Only focus on the most important feedback
  • Be ok with doing a medium job, you don't have to do a perfect job
  • Set boundaries- I decided to do the call alone to reduce my nerves

Startup Update

8-1-20


My spouse and sister are both helping out with the startup (secret Project WT). It's really nice to not be doing all the work.

We each found our places naturally:

  • I'm tech
  • my spouse is finance and credit card processing
  • my sister is marketing/market research

It's blurry because it's still a secret!

Market Research


Whether or not the Project WT is successful, it's a great learning experience for any developer or entrepreneur to build a usable product.

But, if I want a successful product, I can't just build something usable. That's not going to cut it.

Will people actually use Project WT? Will Project WT help someone? Market research is so important.

The idea for Project WT came about from some accidental market research- My sister was perusing a subreddit for a certain occupation. Many of these professionals had the same problem. My sister told me about the problem. We came up with a solution.

Market Research Surveys

As I build the app, we're continuing more market research. We're contacting individuals of this profession to take a survey.

Asking strangers to help you with you're business is uncomfortable. These people are busy and I'm asking them to take a 5-minute survey. I tried to frame it in two ways: By taking the survey they are being 1. altruistic and 2. benefitting themselves. They are helping me out, but also, with their input, I can make a better product that can help them too.

I was excited even only after getting back 2 responses to the survey. We learned two things:

  1. Our intuitions about the needs of our customer were correct.
  2. We learned other needs that we can address with additional features in our product.

Leading A Team


So far my style as a team leader has been to make suggestions but not micromanage. I like to bring up my concerns or ideas, but ultimately let the group decide.

I'm not an expert anyways- this is all new to me. The only reason I'm the leader is because I'm the one who can actually make the product and I'm spending the most time by far on it. But I'm new to business. So I'm open to what my spouse and sister suggest. We are all learning along the way.

I also like to make sure the group stays focused. Sometimes we can get caught up in the details, me included. But I think the details will come to us as we learn. Very few decisions are so important that they need to be completely fleshed out right away. So sometimes I have to say let's make a decision and move on.

I'm still learning how to communicate in a team setting. But I'm prioritizing learning about leadership just as much as coding.

Team Meeting!

As the leader, I don't hold my spouse and sister to any obligation to continue. My sister's semester starts up in fall and I know she'll be really busy. But for now it's really nice to have the help. I'm appreciative of what they've done so far.

We haven't fleshed out payment/equity yet, though I did bring it up before we started to work together. I'm not sure the right way to go about paying them back for their help.

Working with them has given me added pressure to finish the app. I like having them rely on me. It's not too much pressure, but a good amount- like a nice massage.

Side Projects?


I'm putting all my time into the site. I keep wondering if I should make time for other interests like continuing my lucid dreaming project or learning about AI and deep fakes.

But I think it's ok to put them on the back burner for now. My focus is drawn to the startup project, anyways. When my focus starts to wane, I'll then go make time for my other projects.

Update

7-20-20

Deepfakes

Since I last wrote in here, I made another deepfake model. I trained it to about 200k iterations but it started to get worse at one point. So I need to learn a bit more.

Secret Project WT

I've put deepfakes aside to work on a business idea. The business idea is somewhat adjecent to my interest in deepfakes. I don't want to talk about the idea yet. So we'll call it Project WT.

I made a lot of the frontend of Project WT already. Today, I started the backend. I'm using a MERN stack to develop the app.

It kind of feels like I don't know what I'm doing, but I'm going do it anyways.

I Don't Know What I'm Doing

I don't know the "right" way to do things.

Did I structure my app the way I'm supposed to? Did I use this one library the way it's meant to be used?

Not sure. But I can't spend all my time trying to figure out the accepted way of making an app.

I'm just going to make an app- scrappily. For now, 'scrappy' suffices. Scrappy > right.

Also, I'm not entirely alone. My spouse is helping out with the idea where he can:

  • setting up social media accounts
  • researching logistics and payment processing
  • discussing ideas on long walks
  • kisses. sometimes you just need them to get through the day.

My notes/resources to save:

Backend Mern tutorial

Installing Mongo and Starting the new way because the one in the tutorial didn't work:

$ brew tap mongodb/brew
$ brew install mongodb-community
$ sudo mkdir -p /System/Volumes/Data/data/db
$ sudo chown dashiellbarkhuss /usr/local/var/mongodb
$ sudo chown dashiellbarkhuss /usr/local/var/log/mongodb
$ sudo mongod --dbpath /System/Volumes/Data/data/db

Catalina OS update- “mkdir: /data/db: Read-only file system” error

Deepfakes

7/7/20


An Introduction to DeepFakes and Face-Swap Technology

How To Create The Perfect DeepFakes

I kept training my deepfake. I'm surprised it got better even only on 38 source images for Hillary's face.

My First DeepFaceLab Deepfake

6/30/20-7/6/20


Making A (Bad) Deepfake

I made a (very undertrained) deepfake using DeepFaceLabs.

This is my first deepfake using an autoencoder model, the kind of model that DeepFaceLab uses. Other deepfakes I've made with Avatarify used First Order Motion Model.

Why is this deepfake so bad?

I trained this model on a small amount of data and iterations: 38 source images for 500 iterations. That's not enough to produce a quality deepfake.

It's common to train for 100k or more iterations. And use hundreds or thousands of source images.

But my goal with this first deepfake was to get through the DeepFaceLab workflow, not make a great deepfake. Hopefully, this repo can help you get through it as well.

The preview image for my model at 400 iterations.

Please consider donating to DeepFaceLab. This project is open source and a lot of hard work went into it.

Getting Started

I made a repo to help you get started: DFL-Colab-Newbs

Deepfake Readings:

  • Tech Behind Deepfakes #1: link
  • Tech Behind Deepfakes #2: link

DeepFaceLab Guides:

  • DeepFaceLab 2.0 Guide: link
  • Google Colab Guide: link

Deep Learning

6/27/20 - 6/29/20


This was on the MrDeepFakes tutorial for DeepFaceLab with google colab.

I was like what is "the workspace we have setup"? When did we set one up?

The workspace folder is the most important one to note. This is the folder where all the magic will happen. It contains three folders and two video files:

  1. “data_dst”
  • dst = destination: the folder where the original video files will live. When the project is complete, it will include two sub-folders: “aligned” and “merged”. We will talk about these later on.
  1. “data_src”
  • src = source: the folder where the face of the ‘fake’ will live. When the project is complete, it will include one sub-folder: “aligned”.
  1. “model”
  • model: this folder will contain the training model files used for the neural network.
  1. “data_dst.mp4”
  • This file is the destination video where we will swap the fake face with.
  1. “data_src.mp4”
  • This is the source video file where we will collect and gather the face image assets from to train our neural network.

source

DeepFaceLab 2.0 consists of selection of .bat files used to extract, train and merge (previously called convert) which are 3 main steps of creating a deepfake, they are located in the main folder along with two subfolders:

  • _internal (that's where all the files necessary for DFLs to work are)
  • workspace (this is where your models, videos, facesets (datasets) and final video outputs are

Looks like I should read this MrDeepFakes DeepFaceLab tutorial as well as the colab one. A lot is left out in the colab tutorial.

I continued to explore deep face lab without posting all of the links I've been looking at.

Deep Learning

6/24/20 - 6/26/20


I thought deepfakes used gans. But the seem to really use autoencoders.

It depends on what it meant by deep fake. thispersondoesnotexist uses gans. But Deepfacelab uses autoencoders.

How to create a deepfake using GANs ?

The Sparse Autoencoder (SAE) for Dummies

Iteration-

Like any neural network, the autoencoders in Faceswap are trained using backpropagation. The training algorithm feeds a particular image into the neural network and figures out which pixels in the output don't match the input. It then calculates which neurons in the final layer were most responsible for the mistakes and slightly adjusts each neuron's parameters in a way that would have produced better results.

Errors are then propagated backwards to the next-to-last layer, where each neuron's parameters are tweaked once again. Errors are propagated backwards in this fashion until every parameter of the neural network—in both the encoder and the decoder—has been adjusted.

The training algorithm then feeds another image to the network and the whole process repeats once again. It can take hundreds of thousands of iterations of this process to produce an autoencoder that does a good job of reproducing its own input. source

So an iteration must be when a network forward propagates and backpropagates through one input.

I created my own deepfake—it took two weeks and cost $552

What are convolutional neural networks?

Introducing convolutional neural networks (ML Zero to Hero - Part 3)

Convolutional Neural Networks - The Math of Intelligence (Week 4)

Convolutional Neural Networks (CNNs) explained

Dot Product?

The Dot Product gives a scalar (ordinary number) answer, and is sometimes called the scalar product.

source

Vector dot product and vector length

Cross product introduction

Intro to matrix multiplication

Whats the matr

Epochs

One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE. source

Batch Size

Total number of training examples present in a single batch. source

Iterations

To get the iterations you just need to know multiplication tables or have a calculator. 😃

Iterations is the number of batches needed to complete one epoch.

Note: The number of batches is equal to number of iterations for one epoch.

Let’s say we have 2000 training examples that we are going to use .

We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.source

Epoch vs Batch Size vs Iterations

Deep Learning

6/22/20 - 6/24/20


DeepLearning series Ep 1 : DeepFaceLab

DeepFake with DeepFaceLab and Python by Radoslav Nedyalkov (Eng) -not that great, not actually a tutorial

src - a person who will be use for replacement

dst - a person who will be replaced

Autoencoders Tutorial | Autoencoders In Deep Learning | Tensorflow Training | Edureka -too complicated in the beginning but parts were helpful

An Introduction to Neural Networks and Autoencoders -good article!

The most effective architecture for image-based applications so far is convolutional neural network (CNN), and this is exactly what Deep Fakes is using.

Understanding the Technology Behind DeepFakes -also good article!

The diagram above shows an image (in this specific case, a face) being fed to an encoder. Its result is a lower dimensional representation of that very same face, which is sometimes referred to as base vector or latent face

Lower dimensional representation? What does that mean? The image is still 2-D. So what di they mean lower dimension? What are examples of the dimensions? Does it have todo with the size of the vector?

Deep Learning

6/18/20 - 6/21/20

Messy notes and resources:


Ian Goodfellow: Generative Adversarial Networks (NIPS 2016 tutorial)

Words

  • Density Estimation In probability and statistics, density estimation is the construction of an estimate, based on observed data, of an unobservable underlying probabilty density function. (source)
  • agent

    A deep learning agent is any autonomous or semi-autonomous AI-driven system that uses deep learning to perform and improve at its tasks. (source)

Artificial Intelligence Foundations: Machine Learning

What's the difference between transductive learning and inductive learning?

Q-learning has been used for Go. But what about backgammon?

Artificial Intelligence Foundations: Neural Networks

What is the code function in a GAN?

Stochastic Gradient Descent.

Deep Learning

6/15/20-6/17/20


I pranked my sister by zooming her with a deepfake of herself. Watch the video:

This is a link to a Youtube Video of Me Pranking My sister by calling her on zoom with a deepfake of herself

Notes 6/15/20-6/17/20

I've spent a lot of time on the nitty gritty calculus of backpropagation. I think it's time to zoom out and do something more broad like run something on google colabs.

Multiple Versions of Python

Mac OS needs python

DO NOT remove any versions of Python found in the following folders:

  • /usr/bin
  • system/Library

These versions of Python—which should be Python 2.7—are installed by Apple and used by Mac OS and other software to perform some functions. Deleting Python from these directories will break Mac OS and force you to reinstall it on your computer.

Other projects may need specific versions of python

You may have a python project or you may use python packages that require particular versions of Python. Uninstalling those versions would prevent those projects or packages from working until that version of python is reinstalled. For example, Python 3 is a dependency of Numpy; if you uninstalled Python 3, then Numpy wouldn’t work until you reinstalled Python 3.

source

source

source is a bash shell built-in command that executes the content of the file passed as argument, in the current shell. It has a synonym in . (period).

Syntax

. filename [arguments]

source filename [arguments]
Dashiells-MacBook-Pro:~ dashiellbarkhuss$ pip install pyyaml
Requirement already satisfied: pyyaml in /opt/miniconda3/lib/python3.7/site-packages (5.1)

source

Yaml and different versions of python on my computer

I had a problem with yaml saying it wasn't installed when I trying to run

./run_mac.sh --is-client --in-addr tcp://0.tcp.ngrok.io:19547 --out-addr tcp://0.tcp.ngrok.io:15694

I'd get an error

Traceback (most recent call last):
  File "afy/cam_fomm.py", line 4, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

No module named 'yaml' But if I installed

pip install pyyaml

I'd get

Requirement already satisfied: pyyaml in /opt/miniconda3/lib/python3.7/site-packages (5.1)

which python in the terminal gave me

/opt/miniconda3/bin/python

But in the run_mac.sh file I ran which python and I got:

/opt/miniconda3/envs/avatarify/bin/python

So I realize it's the same python environment. So I added yaml to this avatariy environment by add pip install pyyaml to the run_mac.sh file after the environment is set and running it once.

Now it's added but I also need to do the same thing for other packages

Deep Learning

6/9/20 - 6/15/20

Jump To Summary to see main resources used


I still have so much to learn about deep learning. But I think today I will jump out of theory land and do some hands on learning through code.

Reddits For Machine Learning Newbies

  • r/learningmachinglearning I found this reddit helpful yesterday when I had a question.
  • r/MLQuestions This reddit looks good too for newbies.
  • r/deeplearning Haven't checked out the newb scene here yet.
  • For deepfakes:
    • r/deepdream dedicated to art produced via machine learning algorithms
    • r/MediaSynthesis Synthetic media describes the use of artificial intelligence to generate and manipulate data, most often to automate the creation of entertainment.

I really want to get on the MrDeepFakes forum but the forum hasn't been working. Yes MrDeepFakes is a porn site, nonetheless the people on this site are active with deepfakes and know a lot. But the fact that the forum login hasn't worked for about 2 weeks makes me think it's not the best place to learn.

Other Chats/Forums

Do I want to take this udacity course on deep learning? It's 12 hours for 4 months.

Today, I'm going to get into some more hands on stuff: I'm doing the follow along for this tutorial

I'm copy coding this, going through each section and trying to understand what's going on. I'm using google colab to run Jupyter notebook file. (Update: later I switch to Jupyterlab)

W3Schools NumPy

Neural Networks Demystified

I didn't understand what the forward() function was in the GANs tutorial. I found this video about forward propagation. I started watching the whole series it's from called Neural Networks Demystified

Activation functions are mathematical equations that determine the output of a neural network. The function is attached to each neuron in the network, and determines whether it should be activated (“fired”) or not, based on whether each neuron’s input is relevant for the model’s prediction. Activation functions also help normalize the output of each neuron to a range between 1 and 0 or between -1 and 1. source

Sigmoid and ReLU are examples of activation functions.

The Very Scary Calculus-y Part 4 Video

There's so much calculus in part 4 of Neural Networks Demystified. I need to refresh my calculus. Last year, I started taking calculus on Khan Academy to prepare for machine learning, so I'm glad I'm not totally foreign to these concepts. But I need to brush up.

Calculus concepts I reviewed to understand backpropagation:

  1. Derivative as a concept
  2. The derivative & tangent line equations - watched half
  3. Formal definition of the derivative as a limit
  4. Formal and alternate form of the derivative
  5. Partial derivatives, introduction
  6. Differentiability and continuity
  7. Derivative notation review
  8. Chain rule
  9. Power rule
  10. The Chain Rule from wlc.edu I mostly just looked at the first image
  11. Intro to matrix multiplication

I figured out which calculus concepts to review because either the Neural Networks Demystified mentioned them. Or the Khan Academy videos themselves mentioned a concept I was unfamiliar with.

It helped to watch part 4 of Neural Networks Demystified on the companion Jupyter notebook file, because it had a chart of all the variable symbols and what they mean.

I've spent several days trying to understand the math of the 8 minute video on backpropagation. It's sunday and I started on Thursday.

Rewatched Backpropagation calculus

I finished watching the 7 part series Neural Networks Demystified.

Summary of 6/9/20 - 6/15/20


  1. I followed this GANs tutorial. But I stopped coding along when the tutorial got to the part about the forward() function.

  2. I didn't understand forward(). So I watched Neural Networks Demystified which covers forward propagation in depth.

  3. Part 4 of Neural Networks Demystified has a lot of calculus. I had to review these calculus concepts before moving on.

  4. I rewatched Backpropagation calculus from a 3Brown1Brown.

  5. I finished watching the Neural Networks Demystified series but didn't finishing coding along.

Deep Learning

6/6/20 - 6/8/20

3Brown1Brown deep learning series - Neural networks. Superb visuals. He had great content for when I was learning about (brainwave) signal processing as well.

Neural networks by 3Brown1Brown:

  1. But what is a Neural Network? | Deep learning, chapter 1
  2. Gradient descent, how neural networks learn | Deep learning, chapter 2
  3. What is backpropagation really doing? | Deep learning, chapter 3
  4. Backpropagation calculus | Deep learning, chapter 4

Video: A Friendly Introduction to Generative Adversarial Networks (GANs) Tutorial with python follow along on GANs, the machine learning framework used in deepfakes.

Repo for tutorial above

Article: GANs from Scratch 1: A deep introduction. With code in PyTorch and TensorFlow

ML vs AI

For those new to the field of Artificial Intelligence (AI), we can briefly describe Machine Learning (ML) as the sub-field of AI that uses data to “teach” a machine/program how to perform a new task. source

How Machines Learn

The Rise of the Machines – Why Automation is Different this Time

Lecture 13 | Generative Models - only watched a little

Question:

Are there libraries that help machines learn faster by feeding it what another machine already learned?

For example, let's say I have a collection of text messages. I wrote half of them and Shlomo, my bff/lover, wrote the other half. I want a machine to learn to detect which texts I wrote and which Shlomo wrote. In this situation, are there libraries of neural networks that have already taught a machine about the english language? Otherwise my machine with have to learn to recognize so much from scratch.

Is a library like this a thing? Or do you always have to train a machine from scratch?

I posted this question on reddit. I wanted to find a slack or discord to post it on. But I haven't found a good slack or discord for beginners learning ML.

I got an answer

For NLP, your best best is probably scapy, as it comes with three English models ready to deploy. You will still need labels in your training data for who wrote what note, but tokenization, stemming, word vector generation, etc should be ready to go for preprocessing!

Hint: in other prebuilt models, you can also add on to them using transfer learning

Prebuilt models & transfer learning is what I'm talking about I guess!

Not related to deep learing: Tim Ferris/ Noah Kagan Article on business- How to Create a Million-Dollar Business This Weekend (Examples: AppSumo, Mint, Chihuahuas)

Deep Learning

6/4/20 - 6/6/20

Watching this series- this is the first video- How to Make a Prediction - Intro to Deep Learning #1

  • Machine learning vs Neural Network vs Deep Learning are all different things
  • Machine Learning: Traditionally, programming is defining every step for a ccomputer to take to get to an outcome. Machine learning is backwards. You Show the computer the outcome and the computer learns how to get there.
  • Neural Networks: There are a lot of mahcine learning models. One of them is called a Neural Network.
  • Deep Learning: When we use a Neural Network that's not just one or two but many layers deep to make a prediction.

Machine Learning 3 Styles:

  • Supervised
  • Unsupervised
  • Reinforcement

Regression: a machine learning task.

Pandas let's you read your data set. sklearn- machine learning librayr matplot lib- lets you visualize your model and data.

Neural Network- a biologically inspired algorithm that learns to identify patterns in data.

Backpropagation is a technique to train a Neural Net by updating weights via gradient descent.

Deep learning= many layer neural net + big data + big compute

More Deepfakes & Avatarify

6/3/20 - 6/4/20

Avatarify keeps freezing. It's hard for me to figure out why. Maybe it has something to do with Google Colab? I will try to understand mroe about Colab.

From Welcome To Colaboratory:

Colaboratory, or "Colab" for short, allows you to write and execute Python in your browser, with

  • Zero configuration required
  • Free access to GPUs
  • Easy sharing

Whether you're a student, a data scientist or an AI researcher, Colab can make your work easier.

It reminds me of a Jupyter Notebooks. Oh I guess it is a Jupyter Notebook?

What is the difference between Jupyter and Colab?

Jupyter is the open source project on which Colab is based. Colab allows you to use and share Jupyter notebooks with others without having to download, install, or run anything.

-from Colaboratory FAQ

From Welcome To Colaboratory:

Colab notebooks execute code on Google's cloud servers, meaning you can leverage the power of Google hardware, including GPUs and TPUs, regardless of the power of your machine. All you need is a browser.

Could the problem be with Ngrok? What is Ngrok?

What is ngrok?

ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels.

-from ngrok

Network address translation (NAT)

Video: NAT Explained - Network Address Translation

Video: What is ngrok?

  • It lets anyone access your local website.

Ali Replied On Slack

I had posted on the avatarify slack:

My terminal was showing all this around the time it froze. Seemed to go from sending and receiving data to predict returned None and recv_queue is empty

[1591061619.742012] PACK 1.651 (1.518)		UNPACK 0.888 (1.153)
[1591061620.019460] SEND 0.111 (0.116)
[1591061620.803449] RECV 107.846 (44.370)
[1591061620.805253] PACK 1.613 (1.520)		UNPACK 1.687 (1.158)
[1591061621.087406] SEND 0.107 (0.116)
[1591061621.356967] recv_queue is empty
[1591061621.357040] predict returned None
[1591061621.836266] RECV 99.935 (44.233)
[1591061621.837547] PACK 1.561 (1.521)		UNPACK 1.189 (1.159)
[1591061622.072949] recv_queue is empty
[1591061622.073058] predict returned None
[1591061622.088352] SEND 0.133 (0.116)
[1591061622.188220] recv_queue is empty
[1591061622.188299] predict returned None
[1591061622.301631] recv_queue is empty
[1591061622.301709] predict returned None
[1591061622.414096] recv_queue is empty
[1591061622.414171] predict returned None
[1591061622.526833] recv_queue is empty
[1591061622.526892] predict returned None
[1591061622.638287] recv_queue is empty
[1591061622.638335] predict returned None
[1591061622.748865] recv_queue is empty

I just checked the Avatarify Slack and Ali answered:

Looks like the server stopped, did you check that?

Wait a minute, when did I run a server? I mean I knew I did. But I didn't really think about it.

Technical details

The client on your computer connects to the server via ngrok TCP tunnel or a reverse ssh tunnel.

-from the avatarify Colab file

What is TCP/IP?

Well I ran the preview again and I haven't had a problem again with freezing. I think it might just be that the internet over here sucks balls. I've had a lot of lagging but no freezing. My Google Colabs rendered avatar will sometimes lag a whole minute behind my computer preview.

Client Not Starting Again

I'm getting the problem where the client isn't starting again. Maybe it's the internet at the house I'm at.

I ran ./run_mac.sh --is-client --in-addr tcp://0.tcp.ngrok.io:<some #> --out-addr tcp://0.tcp.ngrok.io:<some #> in my client.

It got stuck again:

[1591295183.195187] Loading Predictor
[1591295183.249625] Sending to tcp://0.tcp.ngrok.io:19494
[1591295183.249688] Receiving from tcp://0.tcp.ngrok.io:18279
[1591295183.251305] SEND 0.289 (0.289)

So I opened a new terminal so I could keep the tcp tunnel open and cd'd into my avatarify directory and got the info from the logs:

Client Logs

# cam_fomm.log
$ cat ./var/log/cam_fomm.log | head -100

[1591295183.195119] Loading Predictor

# recv_worker.log
$ cat ./var/log/recv_worker.log | tail -100

[1591295183.249530] Receiving from tcp://0.tcp.ngrok.io:18279

# predictor_remote.log
$ cat ./var/log/predictor_remote.log | tail -100

# send_worker.log
$ cat ./var/log/send_worker.log | tail -100

[1591295183.249461] Sending to tcp://0.tcp.ngrok.io:19494
[1591295183.250705] SEND 0.289 (0.289)

Logs from the server(Google Colab)

# cam_fomm.log
!cat ./var/log/cam_fomm.log | head -100

[1591295131.469881] Loading Predictor

# recv_worker.log
!cat ./var/log/recv_worker.log | tail -100

[1591295136.497368] Receiving on port 5557
[1591295137.498841] recv timeout
[1591295138.500048] recv timeout
[1591295139.501354] recv timeout
[1591295140.502621] recv timeout
[1591295141.503973] recv timeout
[1591295142.505343] recv timeout
[1591295143.506708] recv timeout
[1591295144.508054] recv timeout
[1591295145.509405] recv timeout
[1591295146.510674] recv timeout
[1591295147.512001] recv timeout
[1591295148.513307] recv timeout
[1591295149.514569] recv timeout
[1591295150.515866] recv timeout
[1591295151.517149] recv timeout
[1591295152.518417] recv timeout
[1591295153.519695] recv timeout
[1591295154.520933] recv timeout
[1591295155.521329] recv timeout
[1591295156.522362] recv timeout
[1591295157.523621] recv timeout
[1591295158.524339] recv timeout
[1591295159.525671] recv timeout
[1591295160.526322] recv timeout
[1591295161.527560] recv timeout
[1591295162.528859] recv timeout
[1591295163.530099] recv timeout
[1591295164.531374] recv timeout
[1591295165.532618] recv timeout
[1591295166.533880] recv timeout
[1591295167.535235] recv timeout
[1591295168.536511] recv timeout
[1591295169.537745] recv timeout
[1591295170.538969] recv timeout
[1591295171.540282] recv timeout
[1591295172.541613] recv timeout
[1591295173.542851] recv timeout
[1591295174.543134] recv timeout
[1591295175.544391] recv timeout
[1591295176.545623] recv timeout
[1591295177.546854] recv timeout
[1591295178.548073] recv timeout
[1591295179.549387] recv timeout
[1591295180.550594] recv timeout
[1591295181.551827] recv timeout
[1591295182.553060] recv timeout
[1591295183.553402] recv timeout
[1591295184.037698] RECV 483.494 (483.494)
[1591295185.040406] recv timeout
[1591295186.041745] recv timeout
[1591295187.043012] recv timeout
[1591295188.044255] recv timeout
[1591295189.045485] recv timeout
[1591295190.046714] recv timeout
[1591295191.047975] recv timeout
[1591295192.049220] recv timeout
[1591295193.050551] recv timeout
[1591295194.051841] recv timeout
[1591295194.051943] recv_worker exit

# predictor_worker.log
!cat ./var/log/predictor_worker.log | tail -100

[1591295184.038878] working on {'name': '__init__', 'critical': True, 'id': 0}
[1591295193.297287] predictor_worker error
[1591295193.300892] predictor_worker exit

# send_worker.log
!cat ./var/log/send_worker.log | tail -100

[1591295136.503218] Sending on port 5558
[1591295137.504635] send queue empty
[1591295138.505828] send queue empty
[1591295139.507043] send queue empty
[1591295140.508257] send queue empty
[1591295141.509450] send queue empty
[1591295142.510346] send queue empty
[1591295143.511650] send queue empty
[1591295144.512894] send queue empty
[1591295145.514138] send queue empty
[1591295146.515354] send queue empty
[1591295147.516279] send queue empty
[1591295148.517272] send queue empty
[1591295149.517987] send queue empty
[1591295150.519198] send queue empty
[1591295151.520380] send queue empty
[1591295152.521587] send queue empty
[1591295153.522778] send queue empty
[1591295154.523971] send queue empty
[1591295155.524925] send queue empty
[1591295156.526128] send queue empty
[1591295157.527365] send queue empty
[1591295158.528545] send queue empty
[1591295159.529786] send queue empty
[1591295160.530956] send queue empty
[1591295161.532146] send queue empty
[1591295162.533465] send queue empty
[1591295163.534664] send queue empty
[1591295164.535904] send queue empty
[1591295165.537098] send queue empty
[1591295166.538343] send queue empty
[1591295167.539559] send queue empty
[1591295168.540751] send queue empty
[1591295169.541870] send queue empty
[1591295170.543054] send queue empty
[1591295171.543232] send queue empty
[1591295172.544419] send queue empty
[1591295173.545583] send queue empty
[1591295174.546774] send queue empty
[1591295175.547964] send queue empty
[1591295176.549147] send queue empty
[1591295177.550433] send queue empty
[1591295178.551604] send queue empty
[1591295179.552792] send queue empty
[1591295180.553991] send queue empty
[1591295181.555143] send queue empty
[1591295182.556268] send queue empty
[1591295183.557460] send queue empty
[1591295184.558509] send queue empty
[1591295185.559216] send queue empty
[1591295186.560523] send queue empty
[1591295187.561821] send queue empty
[1591295188.563108] send queue empty
[1591295189.564446] send queue empty
[1591295190.565701] send queue empty
[1591295191.566970] send queue empty
[1591295192.567348] send queue empty
[1591295193.568681] send queue empty
[1591295193.568788] send_worker exit

I think Google Colab is interesting, and might help me learn how to help other people use a remote GPU. However, I couldn't skip all of this hassle by getting an Nvidia GPU. I thought you could not use them with Apple products, but I think that's not entirely true if you have some coding skills. I also wonder how close Ali is to creating a non-Nvidia version of this project? How many other projects use only Nvidia?

!ps

    PID TTY          TIME CMD
      1 ?        00:00:00 run.sh
      9 ?        00:00:02 node
     24 ?        00:00:03 jupyter-noteboo
    116 ?        00:00:00 tail
    124 ?        00:00:02 python3
    256 ?        00:00:00 bash
    261 ?        00:00:03 ngrok
    337 ?        00:00:00 bash <defunct>
    426 ?        00:00:00 ps

Avatarify

6/1/20 - 6/2/20

avatarify is an open source "deepfake" program that you can use live on Zoom. For example, you can Zoom into your meeting as Einstein so people will take you seriously.

I followed the instructions for installing avatarify on my mac and then I followed the instructions to run the Google Colab.

Notes on following the readme and Google Colab instructions:

  • Make sure you're doing the Colab in your own copy. I'm not sure how the Colab works, but I think it's sort of like Google docs- if you make changes to the document, you changed it for everyone. So before you even start playing with the Colab you should copy to drive and run your copy of the Colab.

  • Remember to restart the terminal after installing miniconda before running bash scripts/install_mac.sh. Otherwise you could get an error: -bash: conda: command not found.

Freezing Up

When I run avatarify in Google Colab, sometimes nothing happens when I try to start the client.

I'll run the script in my terminal:

./run_mac.sh --is-client --in-addr tcp://0.tcp.ngrok.io:<# collab it gives you> --out-addr tcp://0.tcp.ngrok.io:< # collab gives you >

And I get back something like this:

[1590494901.630968] Loading Predictor
[1590494901.735869] Receiving from tcp://0.tcp.ngrok.io:15702
[1590494901.735815] Sending to tcp://0.tcp.ngrok.io:16110
[1590494901.747087] SEND 3.435 (3.435)

And nothing happens. A lot of things seem to cause this problem: another process might still be running from last time or you might have accidentally changed some code. To stop this from happening, I start all over. Sort of like refreshing everything.

Here's how I refresh everything.

  1. BEFORE YOU START: First, make sure you're doing everything in your own version of the avatarify Colab. I'm not sure how Colab works, but I think it's sort of like Google docs- if you make changes to the document, you changed it for everyone. So before you even start playing with the Colab you should copy to drive and run the Colab on your copy.

  2. When you run into a problem (like I did with the client not starting), revert to the original state:

  3. And terminate any running sessions

  4. On your computer, start a new terminal just in case.

Preview Freezing

When I do get the preview to appear after running ./run_mac.sh --is-client.... it will start working and then freeze. As seen here:

This is where I am stuck.

Looking at this, I realize Einstein and I both have ample flyaways. Does this mean I'm a genius too?