Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔴💜evaluate(pivot game, BayesENT) #3

Open
hyunjimoon opened this issue May 18, 2024 · 8 comments
Open

🔴💜evaluate(pivot game, BayesENT) #3

hyunjimoon opened this issue May 18, 2024 · 8 comments

Comments

@hyunjimoon
Copy link
Contributor

hyunjimoon commented May 18, 2024

preparing for Charlie test (monday) and Scott test (tuesday)

  • upload notebook explaining simulation tool in github
  • run automated hypothesis plotting
  • make this continuous integration

OUTC stands for optimism, uncertainty, tolerance of risk-cash, cash.

peek of notebook:
revenue and profit distribution of angie's product (ECON, PC) X market (ENT, OM ) fit

Image

Image

Image

@hyunjimoon hyunjimoon converted this from a draft issue May 18, 2024
@hyunjimoon hyunjimoon changed the title 🔴 💜evaluate(OUTC, SC) 🔴💜evaluate(OUTC, SC) May 18, 2024
@hyunjimoon
Copy link
Contributor Author

hyunjimoon commented May 20, 2024

point out there are two updates: product/market + belief. belief happens before predict range

Image

Image

@hyunjimoon
Copy link
Contributor Author

hyunjimoon commented May 20, 2024

tl;dr

  • I started with parameterizing eight (profit and revenue for four product and market pair). let alone the number of parameters, this was complicated as i needed to simulate two environments (that generates real cost and revenue).
  • after observing my 1 year of product market pivot path, i came up with new parameterization: $\mu_{deliver}, \mu_o2e, \mu_p2e$ which represents eight parameter with three (mean delivery and mean difference between two product and two market).
  • updating product and market itself should be separated from updating belief. there are learning on cost and revenue each (not profit), but the final decision to pivot is decided by comparing predicted profit (low and high) with observed profit (profit_obs).
  • remaining two q
  • Q1: how to connect signal on cost and revenue with profit_obs
  • Q2: get charlie test on optimism, overconfidence, rational optimism from the table
  • Q3: how relation between p(mbar) and pbar(m) be and how it'd affect pivot

Image

Image

parameter State0 Revenue or Cost Signal1 State1 Revenue or Cost Signal2 State2 Revenue or Cost Signal3 State3 Revenue or Cost Signal4 State4
$\mu_{p2e}$= $0.5 \times (\underset{p_b}{\max} p_b(m_r) - \underset{p_b}{\min} p_b(m_r))$

$posterior(\mu_{p2e})$ =
$prior(\mu_{p2e})$
- .05×(# of CP+ - # of CP-)
+ .05×(# of CE+ - # of CE-)

CP: signal on PC product's cost
CE: signal on ECON product's cost
.1 CP-1: insightful Vikash's summer school
CP-2: Vikash shared gen-finance demo

CE-1: equipped micro/macro/econometrics
CE-2: data + digital entrep. expertise from NUS collaboration
.1
−.05×(0−2)
+.05×(0−2) = .1
CP+3: BEC conference audience mostly encouraged on reading literature
CE+3: adopted critical standpoint on social science without integrative design thanks to Abdullah's class
.1
−.05×(1−0)
+.05×(1−0)
= .1
.1 CE+4: (internal) conflict with scott's methodology
CE-5: good collab. w/ NUS team+ Amir on econometrics model

CP-4: Coding experience in transportation class

CP-5: AI clockspeed faster and with to make (+relative's plan to corporate ai company in boston)
.1
−.05×(1−1)
+.05×(2−0)
=.2
$\mu_{o2e}$ =
0.5 $\times (\underset{m_b}{\max} p_r(m_b) - \underset{m_b}{\min} p_r(m_b))$

$posterior(\mu_{o2e})$ =
$prior(\mu_{o2e})$
+ .05×(# of RO+ - # of RO-)
- .05×(# of RE+ - # of RE-)

RO: signal on OM market's revenue
RE: signal on ENT market's revenue
0


RE+1: Positive feedback from Scott
RE+2: NUS collaboration leads to data access

0
+.05×(0−0)
−.05×(2−0)
= -.1
RO-1: POMS editor was unfamiliar with Charlie

RE+3: Insights from Bayesian conference, esp. Zenger's interest in my connection on his type1-2 with prob.comp lab's kidnapped robot



-.1
+.05×(0−1)
−.05×(1−0)
=-.2
-.2 RO+1: Toni Antonio, Kris Ferrira (TOM seminar) showed interest and suggested simulation for belief modeling

RE+4: Support from Josh Lerner
RE+5: Collaboration with Amir
RE+6: opportunity to use sim.tool for DesignX workshop
−.2
+.05×(1−0)
−.05×(3−0) = -.3
$\mu_d$: delivery


$posterior(\mu_{d})$ =
$prior(\mu_{d})$
+ .2×(# of S - # of S-)
.7 .7 .7 S-1: Jinhua+Charlie's feedback on my delivery .7
+.2×-1 =.5
.5
Optimism: $\underset{m_b,; p_b}{max ;}p_b(m_b) - p_r(m_r)$ .8-.5 = .3 .9-.5 = .4 1-.5 = .5 .8-.5 =.3 1-.5 = .5
Overconfidence: $E_{m_b,; p_b} p_b(m_b) - p_r(m_r)$ .7-.5 = .2 .2 .2 0 0
Rational Optimism: $\underset{m_b,; p_b}{max ;}p_b(m_b) - E_{m_b,; p_b} p_b(m_b)$ .3 -.2 = .1 .4-.2 =.2 .5-.2 = .3 .3-0 = .3 .5-0 = .5

Q3 photo
Image

@hyunjimoon
Copy link
Contributor Author

hyunjimoon commented May 21, 2024

charlie suggested implanting my story in more general framework. based on this i'll start coding low-high bar model.

This company's pivot journey from a no-AI B2C focus to an AI-driven B2B strategy is encapsulated by the evolving belief states on three critical parameters: the cost preference of AI over manual production ($\mu_{m2a}$), the revenue preference of B2B over B2C markets ($\mu_{f2s}$), and the baseline profit ($\mu_{pb}$). Initially, the company believed AI had marginally higher costs and B2B offered slightly better revenue potential. However, signals such as reduced AI development costs, increased maintenance costs for no-AI products, and positive feedback from B2B clients led to an updated belief that AI and B2B were more favorable. This shift is visually captured in the updated parameter plot (BTC: Belief by Time and Component), showing $\mu_{m2a}$ increasing from 0.1 to 0.25, $\mu_{f2s}$ from 0 to 0.25, and $\mu_{pb}$ stabilizing around 0.45. The profitability belief state updates plot (BTS: Belief by Time and Space) illustrates how profitability beliefs for each product-market combination evolved, highlighting AI in B2B markets as the most profitable combination. The accompanying table provides a detailed account of the signals received at each state and how they influenced the parameter updates. By modeling measurement uncertainty, which is a function of clockspeed (as faster changes increase uncertainty), the update of $\mu$ would not monotonically increase or decrease to the true value (as in BTC plot), allowing for more accurate and dynamic adjustments.

image

image

notable.

  • gpt also asked me ways to reduce ai costs and increase b2b sales (as if it read my mind to pivot to ai ->b2b market)
  • relative pessimism on baseline profit (average) can be overcome by optimism on specific product and market segment
  • process: claude and gpt

@hyunjimoon hyunjimoon changed the title 🔴💜evaluate(OUTC, SC) 🔴💜evaluate(BTCS, SC) May 23, 2024
@hyunjimoon
Copy link
Contributor Author

hyunjimoon commented May 23, 2024

angie showed this (video in preparation) to get scott's desirability test:

1.📚 read two papers (theory-based entrepreneurial search + ederer manso)

  • ederer manso has similar experimental setup letting participants change either product mix or market (location)
    image

  • theory-based entrepreneurial search (may 2024) "develop a Bayesian model where entrepreneurs update their beliefs as they conduct entrepreneurial search. We find several optimal behaviors for theory-based entrepreneurs such as reverting to a previous strategy after finding a relatively poor strategy and continuing to search after finding a relatively good strategy, which are missing when entrepreneurs lack such a theory-based approach" figures from paper:
    image

  1. 🧠low and high bars depend on entrepreneur's prior Good initial results may encourage further exploration in a promising environment; agents' beliefs on environment shape whether good news encourages further search or exploitation. Beliefs about the overall opportunity and relative value of product-market combinations are key.
  • prepare (below) charlie's desirability test
  1. 📈 Interesting comparative statics to examine include experiment costs, initial conditions, getting stuck in suboptimal combinations, and how different value landscapes affect search behavior.
  • get charlie's desirability test
  1. 👥 possibility of connecting this with solo vs duo founding teams isn't straightforward but duos may explore more due to elimination of bad ideas via discussion. Miaomiao and I suggested "conflicting literature between tightness of prior of solo founder VS duo founder. phenomenally, solo founder's tight prior comes from his/her behavioral stubbornness whereas duo founder's tight prior comes from internal validation via persuasion" testing these components with simulation parmaeter (e.g. baseline profit $\mu_{a}$, product gap $\mu_{b}$, market gap $\mu_{c}$,$\sigma_a, \sigma_b$, $\sigma_c$ would be interesting in tracing solo vs duo's resistance to broad external signals.
  • sync prior w/ charlie

5.💡tying the model to empirical facts is crucial for persuasiveness.

  • sync prior w/ charlie on NUS collaboration developed here - meeting tonight

different hypothesis for 2. 🧠
Angie is testing gpt's table with the hypotheses, simulation setup, and additional details on how to set the low and high bars for the heuristic decision rule:

Hypothesis Simulation Setup Low and High Bar Heuristic
1. Theory-based entrepreneurs who receive good news will be more likely to continue searching compared to Practice-based entrepreneurs. - Model Theory-based entrepreneurs with two potential distributions (working and alternative theory) and Bayesian belief updating
- Model Practice-based with a static mixture distribution
- Vary initial priors and cost of search
- Set low bar at the 25th percentile and high bar at the 75th percentile of the working theory distribution
- If observed profit is lower than the low bar, pivot to the alternative market
- If observed profit is between the low and high bars, pivot to an alternative product
2. Theory-based entrepreneurs will show more variability in beliefs about the overall opportunity (i.e., the distribution mean). - Track shifts in beliefs about the overall opportunity for both types of entrepreneurs - Not applicable
3. Upon finding a disappointing strategy after initial good results, Theory-based entrepreneurs will be more likely than Practice-based entrepreneurs to revert to a previously found good strategy. - Monitor strategy reversion after disappointment for both types of entrepreneurs - Set low bar at the 25th percentile and high bar at the 75th percentile of the alternative theory distribution
- If observed profit is lower than the low bar, revert to the previous best strategy
4. Theory-based entrepreneurs will value advice and knowledge from experienced agents like VCs and mentors more than Practice-based entrepreneurs. - Introduce an option for entrepreneurs to receive advice that reveals the true distribution
- Track the value placed on this advice by both types of entrepreneurs
- Not applicable
5. The difference in search behavior between Theory and Practice-based entrepreneurs will be largest when initial priors are highly uncertain. - Vary initial priors from highly certain (e.g., 95% confidence in the working theory) to highly uncertain (e.g., 50/50 split between working and alternative theories)
- Compare differences in search behavior between entrepreneur types across different levels of initial uncertainty
- Set low bar at the 25th percentile and high bar at the 75th percentile of the working theory distribution when initial priors are certain
- Set low bar at the 25th percentile of the working theory distribution and high bar at the 75th percentile of the alternative theory distribution when initial priors are uncertain

@hyunjimoon
Copy link
Contributor Author

hyunjimoon commented May 27, 2024

need charlie test on four (?)

1. desirability of pivot game product

from pivot_game product description,

QD: how much do you need this pivot simulator?
this depends on the clockspeed of the domain educator is in. clockspeed is largely determined by organization rigidity, technology/innovation push, product's bit and atom ratio, customer pull, system complexity, branding, and regulation. domain with slower clockspeed will place higher premium on my simulator as they will be more discrete in making a pivot move. Usually deeptech has closer clockspeed and my assumption is clockspeed will increase in order of engine < designX < martin trust center. Hence scenario analysis with be more desired. My plan on theory side is to bridge Charlie's clockspeed (Fine, 2000) with Scott's appropriability (Gans and Stern, 2017) during summer.

2. clockspeed

Image

3. clockspeed, strategy, operation

Image

4. clockspeed -> endogenous appropriability

summarized key from scott's Endogenous Appropriability.pdf

  1. Appropriability, or a firm's ability to capture value from its innovations, is not just determined by exogenous factors like IP rights, but can be shaped by a firm's strategic choices. investing in control is akin to investing to compete for the market whereas investing in execution is competing in the market by establising dynamic capabilities This aligns with clockspeed emphasis on the role of managerial decision-making in navigating dynamic environments(?).

  2. Firms face a key trade-off between investing in "control" strategies (like IP protection) versus "execution" strategies (like lead time advantage).

  3. Industry clockspeed influences this trade-off. In fast-clockspeed industries, execution is favored over control due to rapid change.

  4. A firm's position in the value chain also impacts its appropriability strategy. Upstream firms may rely more on control, while downstream firms focus on execution. clockspeed's emphasis on a firm's role in the "value chain" is relevant(?)

  5. Endogenous appropriability highlights interplay between a firm's strategic choices and the competitive dynamics of its industry. This systems view aligns with "double helix" of industry evolution driven by firm strategies.

process

@hyunjimoon
Copy link
Contributor Author

model that captures the trade-off between optimism and understanding in early-stage startups. Using a simulation model, it demonstrates how self-optimism serves as an engine for executing experiments, which gradually transitions to a better understanding of the environment. The model employs an EM algorithm to frame experiments as iterations of predict, observe, update belief, and environment, with four layers: belief, product's expected profitability, product-market, and market's signal. The interaction between these layers through key actions (predict, observe, update A, and update L) reflects the feedback loop between beliefs, products, and market outcomes, driving continuous adaptation and optimization of product-market strategies.
image

@hyunjimoon
Copy link
Contributor Author

No description provided.

@hyunjimoon
Copy link
Contributor Author

Image

@hyunjimoon hyunjimoon changed the title 🔴💜evaluate(BTCS, SC) 🔴💜evaluate(pivot game, BayesENT) Jun 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🧬 Benchmarking Simulation Tool
Development

No branches or pull requests

1 participant