Monorepo migration - bring "Data Magic" into the primary repo

[futuresoup]

It's time to get the dirty Data Magic repo cleaned up and merged into our primary repo.

@ArLough is leading this, with advisory assistance from @bigolboyyo, @justBanks, and @shreeup

Tasks:

• Copy over the Flask back-end (App folder)
• Migrate Issue tickets over
• Get the Flask app deployed and API accessible
  • Option A: using onrender.com via @shreeup's instructions in comments below (requires paid subscription)
  • ?? Research a free option or a paid option with a free open source license ??
• Create a production branch and deployment workflow
• Rename "Data Magic" repo to "Playground" for running community experiments and testing @Yinan0409's AI model
• Update documentation
• File ticket to set up Replit

[ArLough]

Thinking ahead to deployment and for discussion purposes, here is one article I was reading about a possible way to do it
https://testdriven.io/blog/flask-render-deployment/

There probably is a way to do it with our current host (Netlify I'm pretty sure), but the generalities in the article are still true

There's also this slightly different approach which is in the context of Netlify https://testdriven.io/blog/static-site-flask-and-netlify/

[futuresoup]

@ArLough very helpful thanks!

I'm also open to alternatives to using something besides Flask or Netlify for this project if you or anyone else wants to propose anything.

The Flask decision was based on it's "non-opiniated" simplicity and the fact that the site started with MKDocs (Python) and it involves AI/ML (Python) so it made sense to use Python for everything.

I don't imagine switching to Docusaurus (Node / React) changes anything when it comes to Flask, but if it does let me know, as I am not a developer.

I also don't know if Flask has limitations as the project scales. If so, please share those opinions.

As for Netlify - I happen to like working with it because it's so easy to administer and has some great features that would be a pain to do with AWS. But it does cost money compared to GitHub pages, and is probably more expensive than AWS.

[futuresoup]

@shreeup has been playing around with deploying "Data Magic" to Vercel.

If it's a better fit than another host, that's awesome.

My main concern is price - would I have to pay for both Netlify & Vercel? That starts to add up on top of the other project costs.

I suppose whatever we use to host this thing will have a cost to it beyond Netlify? Or can we consolidate?

[shreeup]

this is the error i received when deploying to vercel. I am trying to make use of free tier
No framework detected. Default Project Settings:

• Build Command: npm run vercel-build or npm run build
• Development Command: None
• Install Command: yarn install, pnpm install, or npm install
• Output Directory: public if it exists, or .
  ? Want to modify these settings? [y/N] y
  ? Which settings would you like to overwrite (select multiple)? None
  🔗 Linked to shreeupadhyaya/y (created .vercel and added it to .gitignore)
  🔍 Inspect: https://vercel.com/shreeupadhyaya/y/4cb5SJB8z8LnfBnRk6AbwA9zxX3L [2s]
  Error: A Serverless Function has exceeded the unzipped maximum size of 250 MB.
  shree@rahuls-mac-mini ClimateTech-data-magic % vercel
  Vercel CLI 30.1.2
  🔍 Inspect: https://vercel.com/shreeupadhyaya/y/HJeM4KBgJNEAD14jbYBrfuNHUqf7 [2s]
  Error: A Serverless Function has exceeded the unzipped maximum size of 250 MB.

I am trying to find alternatives

[shreeup]

i am going to delete venv folder from data-magic github repo (https://github.com/climate-tech-handbook/data-magic/tree/main/venv) . i think it is not really used and make the project heavy, in an effort to make it easier to deploy

[ArLough]

My main concern is price - would I have to pay for both Netlify & Vercel? That starts to add up on top of the other project costs.

I suppose whatever we use to host this thing will have a cost to it beyond Netlify? Or can we consolidate?

Netlify does have serverless functions as an offering, but I'll have to look into it and see if it fits our use case.

[ArLough]

I don't imagine switching to Docusaurus (Node / React) changes anything when it comes to Flask, but if it does let me know, as I am not a developer.

I also don't know if Flask has limitations as the project scales. If so, please share those opinions.

I don't think the Docusaurus switch will affect anything. Flask itself has a reputation as best for side projects and smaller web apps. But, I think it should work at least in the immediate future. There are alternatives that are still Python based like Django and FastAPI that are meant for more production sites and larger websites. But, I think we should be able to make that switch in the future if needed. FastAPI in particular looks very similar to Flask/

[shreeup]

I was able to deploy docker image to render platform,
https://climatetechdatamagic.onrender.com/hello
but post endpoints are not really working.. will have to figure this out
Error

Traceback (most recent call last):
Jun 14 09:47:31 PM File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2528, in wsgi_app
Jun 14 09:47:31 PM response = self.full_dispatch_request()
Jun 14 09:47:31 PM File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request
Jun 14 09:47:31 PM rv = self.handle_user_exception(e)
Jun 14 09:47:31 PM File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request
Jun 14 09:47:31 PM rv = self.dispatch_request()
Jun 14 09:47:31 PM File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request
Jun 14 09:47:31 PM return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Jun 14 09:47:31 PM File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1890, in ensure_sync
Jun 14 09:47:31 PM return self.async_to_sync(func)
Jun 14 09:47:31 PM File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1911, in async_to_sync
Jun 14 09:47:31 PM raise RuntimeError(
Jun 14 09:47:31 PM RuntimeError: Install Flask with the 'async' extra in order to use async views.

Steps:

Build docker image push to docker hub

docker image build -t datamagic_docker .
docker run --platform linux/amd64 -p 5000:5000 -d datamagic_docker
docker login
docker tag datamagic_docker shreeupadhyaya/datamagic_docker
docker push shreeupadhyaya/datamagic_docker

deploy image from docker hub to render
https://render.com/docs/deploy-an-image

[ArLough]

I experienced the same async issue on my local machine. It was resolved by doing 'pip install asgiref' for me

https://stackoverflow.com/questions/70321014/runtimeerror-install-flask-with-the-async-extra-in-order-to-use-async-views

[shreeup]

Ok now I am able to invoke post methods in hosted platform.

This is image from docker logs which show that post endpoints invocation was successful and returned 200.

I am also attaching screenshot of postman returning 200 for one of endpoint