Agreeing on a deliverable (web app?) for parking demand index

[katoss]

Hey,

so in yesterday's meeting we talked about in which form we could create a deliverable from the parking demand index for the city.

Probably a small interactive web-app would be nice, if there are interested/experienced team members, and if it doesn't take too much time. Otherwise we could of course also do notebooks as deliverable.

We talked about implementing it in shiny (R-based, which @dietrichsimon has experience in), or with FastAPI (python based, @akashrajkn has experience).

Anyone who's interested, feel free to join the discussion :)

[katoss]

Hey, I've been trying to get an overview of stacks we could use to create a web app. What about using plotly's Dash, with flask and heroku for deployment? @akashrajkn dash seems to be native to flask and not fastapi, do you think something would speak against using flask?
For heroku, their free plan ends end of november, but I'm keeping my hopes up for a free student version.
I have never used Dash, so I don't know if that's be best thing to use, but to me it sounds like the equivalent of shiny and thus quite good for our context.
Any thoughts, experiences, feedback very welcome :)

[mpadge]

Heroku in this current transition-away-from-free phase is too unstable. It might settle down again at some stage, but i would recommend not starting anything Heroku-based at the moment.

[katoss]

@mpadge I agree that heroku might not be the most stable solution at the moment.. Do you know any good, free alternatives?
I saw that there will be a student plan for heroku, which I am probably eligible for (one can apply from tomorrow on) and which could be sufficient for a proof of concept.
It might also be possible to get some funding in case we don't find anything for free, I'll have to check that. However, a free version would be preferable

[mpadge]

I don't know any good alternatives, alas. But it seems they've learn a lot in the recent, sometimes disasterous re-shuffle, so maybe the student plan might be a good and sufficiently stable option?

[mpadge]

Update: https://news.ycombinator.com/item?id=33077118 has some good links and heroku alternatives

[katoss]

thank you @mpadge ! I skimmed through the discussion, there are many things that are self-hosted, that could complicate things. Maybe fly.io could be an option but I have to look more into how it actually works.

An update: I just got my heroku student account confirmed.
With that account I can spend the worth of 13$ per month, which means

"The maximum applicable credit is $13/month, which is enough to cover one month of the Eco Dynos plan ($5 for 1000 dyno hours/month), one Mini Heroku Postgres database ($5/month), and one Mini Heroku Data for Redis database ($3/month)."

I don't exactly know what we need for our project, but it might be enough.

What do you think?

[mpadge]

sounds like plenty enough, i'd say

[akashrajkn]

I've put my thoughts below in case we decide to use python instead of Shiny (R)

Heroku alternatives (in case we decide to use python - flask/fastapi).

• AWS eb (link - flask app deployment). Using the aws free tier, we could host it for a year. In terms of configuration, it is about the same amount of work as deploying a heroku app.
  • A variation is to just EC2 containers. Using free tier, we could load up a micro instance, that hosts the webapp. This gives total control over the application.
• Python anywhere: link. I've not tried this before, but we could look into it.
• Google cloud engine: link. Also similar to aws deployment. On sign up, we get 300 credits which could last for 5-6 months based on the usage.

I prefer AWS :)

Regarding Plotly, dash and flask

dash seems to be native to flask and not fastapi, do you think something would speak against using flask?

You are right, dash is tightly coupled with flask. Fastapi allows you to mount wsgi applications like flask within it. In theory it should work, but it only complicates things (it is just simpler to use flask only). here's a starter project which uses dash and fastapi (github link)

For our usecase, it is quite simple, so we could just stick to Flask and not complicate things. What do you think?

[operte]

What about asking @friep for ideas? Perhaps CorrelAid already has some kind of platform that we can use.

What about using GCP? We already have an account there, if it's cheap enough it could be an option.

We've been using plotly in the analysis notebooks, so if that works well with flask that would be great :) But of course it wouldn't be a lot of trouble to change plotting packages.

We should also think about how to organize the repo and the code. I can start by transforming the notebooks into functions. Then we can decide how to handle data (process the files live? batch-process them once a day or week?).

[katoss]

@akashrajkn and @operte, thank you for sharing your thoughts!

@operte I will forward the issue to @friep to get their take on deployment alternatives.
And Plotly, dash and flask should work well together as far as i understood, so no need to change plotting packages :)
And yes, organizing the repo and code is another big topic! I don't have much experience in that yet but am very interested. Do you have any preferences on how to best proceed? Maybe opening a dedicated issue for that?

[pr130]

Hey everyone,

thanks for pinging me on Slack @katoss .

as a general tip - but i think everyone has this already on their radar: keep it simple. it might be tempting to build a full-fledged application or dashboard but especially the latter can quickly escalate and tends to result in a "never ending" project scenario.
Also, in general, we (as in CorrelAid) do not host web apps long-term (i simply don't have the capacity to take care of the maintainence :( ). we have done "temporal scenarios" in the past which means I still host an application 2.5 years later :D so i'd like to avoid that.

I do understand that making a small web app has benefits for the presentation of your results. From your POV as participants, I'd suggest weighing the benefits of making a web app (more possibilities for presentation & interactivity -> potentially positive impact on project impact within the City) against the benefits of doing something more static (-> you could simply send it as an email / upload somewhere. also you as participants could more easily reference it in your CV / on your personal website).
In summary, I tend to recommend doing something with notebooks / RMarkdown / Quarto but if you want to, of course go for it! :) (that's not meant to sound passive aggressive ;)).

Deployment wise:

• R/Shiny: we have a premium account on shinyapps.io that you could temporally use. What others have done is to develop with their own free plan and then the organization gets the 10$ plan and there's a final handover.
• In general, i'm not a big shiny fan though 🙈 and especially deployment wise, the Python frameworks are much more flexible.
• We have servers on Azure though, so if you need a small instance, I could give you access and you set it up (but it'd then up to you to maintain it!). In general, i'd like to avoid that though due to the maintainence problem (see above). we don't have AWS or GCP virtual machines (Microsoft had the most attractive NPO program ^^)
• If you end up hosting something, I'd recommend to communicate a clear deadline until when you'll support it :) maybe they'll then take it over or host it themselves. :)

Unfortunately, that's all i can offer. 😬

[operte]

You make some very good points. In my view, the final goal is for the City of Paris to have some useful tools to build from. The responsibility of improving the code or maintaining any web apps should be on them. As such, we should either send them a static dump of code+visuals that they can then continue in whatever way they want, or we could build a web app ourselves, but first aligning with them what would be their preferred platform, since they would be the ones maintaining it.

I would like to get some more experience in working on web apps, though, so if we have time I would definitely like to help build something very simple :)

[katoss]

Thank you for your thoughts @friep and @operte !

I agree, a web app is not absolutely "necessary" and could become a lot of work. Nevertheless, like @operte, I'd be interested in getting some more experience in building a web app :)

I would suggest that we prepare a MVP deliverable, e.g. a notebook. Some brainstorming: values for the city would be a metric that calculates parking demand, integrates public datasets, and visualizes on a map with IRIS, in python (which they prefer). They can use the metric to see if it is in line with other findings, or even give new insights. they can also reuse the method /code as inspiration for other metrics that visualize something based on IRIS sectors.

The web app could be more of an extra, because we feel like trying it. We can see how far we get, and if we manage to do so we can host it for a few months as a proof of concept. We can then ask the city if they would like to continue deploying it after that time on their own terms. It could be interesting for their new data science team as an example.

A general/technical question: should we have two git repositories, one for notebook + other static outputs, and one for the web app ? mixing the two would probably mess up the web app, right? #beginnersorry:)

[Liyubov]

Hi Kati,
sorry to reply late to this.

A general/technical question: should we have two git repositories, one for notebook + other static outputs, and one for the web app ? mixing the two would probably mess up the web app, right? #beginnersorry:)

To your question usually it was sufficient just to have two folders, not the repos.