Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Live hosted instance :) #29

Closed
mindthegab opened this issue Jul 15, 2023 · 11 comments
Closed

Live hosted instance :) #29

mindthegab opened this issue Jul 15, 2023 · 11 comments

Comments

@mindthegab
Copy link
Member

At the cost of putting more work on @maoo backlog, I really would love to see a "live demo" link in the readme as well as on the FINOS website.

@DovOps is the project at the right stage? @maoo maybe something we can work on in August?

BTW, this is amazing :)

@rocketstack-matt
Copy link
Member

Question on this one - what hosting infra is available? Using the existing docker compose setup this should be pretty straight forward. Tag: @karlmoll FYI

@karlmoll
Copy link

thanks @rocketstack-matt - i will bring this conversation up internally as we get this meetup figured out. appreciate the tag!

@maoo
Copy link
Member

maoo commented Dec 21, 2023

Hi @rocketstack-matt - thanks for raising this, I was about to open the same conversation, as I quickly discussed with @mindthegab yesterday and had a follow up with @DovOps on my todo list.

I think that we can easily spin up the Docker Compose, and I'll discuss the technical details below, but first I'd like to better understand the purpose of this environment, its users and the data flow.

For example, we want to spin up a demo.traderx.finos.org frontend, load the system with mock data, which gets re-bootstrapped every 48 hours, with a user (credentials trader/trader) that can access and play a bit with the system; something like demo.waltz.finos.org . I'm not sure if this is what you had in mind, or you were thinking about something different.

Now for the technical part.

We usually run project infrastructure on AWS, but we're open to discuss other providers if you prefer; I must admit, I've never deployed a Docker compose in production, as I always considered it a development/staging tool; however, it seems that things have changed.

Searching for "Docker Compose on AWS" I found this, which seems to be a decent starting point (and uses ECS, which costs way less than EKS).

It would help if the docker compose definition defines cpu and memory reservations/limits, in order to get a better idea of the costs.

That's the bare minimum I guess; after that, we'll probably have to build a bunch of scripts/configurations around it to manage automated backups and restores, some AWS configuration, DNS configurations and that's it.

Eager to hear your thoughts!

@rocketstack-matt
Copy link
Member

Hey @maoo I guess the purpose I'll let @mindthegab and @DovOps opine on, but your description seems inline with my assumptions of what they'd be after.

I would agree with you regarding Docker Compose in a real prod environment, but think it would probably do the job for this demo instance. I can look to add in the compute requirements, but I know from running this up in Codespaces that it requires an environment with at least 32GB for everything to run up successfully.

Do you think we would even need backups? My assumption is we'd run this purely as a demo and any data loss would be acceptable. If we want something more than that then we may want to look at hooking it up to a DB service which could handle the backup, but I guess we're back to your first question of what is the actual use case.

FYI I had flagged this for @karlmoll as we were discussing a meetup he's planning on the new year which TraderX will be at least a large part of.

@maoo
Copy link
Member

maoo commented Dec 21, 2023

Hey @maoo I guess the purpose I'll let @mindthegab and @DovOps opine on, but your description seems inline with my assumptions of what they'd be after.

I would agree with you regarding Docker Compose in a real prod environment, but think it would probably do the job for this demo instance. I can look to add in the compute requirements, but I know from running this up in Codespaces that it requires an environment with at least 32GB for everything to run up successfully.

Ok, that's already a good indication. I'll try to find some time tomorrow to start playing with it.

Do you think we would even need backups? My assumption is we'd run this purely as a demo and any data loss would be acceptable. If we want something more than that then we may want to look at hooking it up to a DB service which could handle the backup, but I guess we're back to your first question of what is the actual use case.

Yes, agreed on everything. Is there any data bootstrap process already in place? If so, we could start an ECS scheduled task that wipes-out / bootstraps data every X hours and we keep it simple for now; in the future we can delegate to AWS RDS.

FYI I had flagged this for @karlmoll as we were discussing a meetup he's planning on the new year which TraderX will be at least a large part of.

Thanks for that! It's always good to have another pair of eyes 😄.

@rocketstack-matt
Copy link
Member

rocketstack-matt commented Dec 22, 2023

Yes, agreed on everything. Is there any data bootstrap process already in place? If so, we could start an ECS scheduled task that wipes-out / bootstraps data every X hours and we keep it simple for now; in the future we can delegate to AWS RDS.

The initialSchema.sql file contains both the schema and initial data load. I think the easiest thing would be to simply schedule to restart the entire instance every 24 hours.

@maoo
Copy link
Member

maoo commented Dec 22, 2023

Hey @rocketstack-matt - I did a first run of tests, and it didn't go really well to be honest:

  • The Docker Compose ECS integration project was archived last month - https://github.com/docker/compose-ecs ; still works, but I'm not sure how long it will be shipped in the Docker CLI
  • I had to publish images to ECR, in order to get it to work; I guess this is code that will come in handy if/when we want to run it on AWS, maybe as GitHub Actions ; see gist below.
  • I run a docker-compose with just the database and had to strip off the volumes part (not supported by the ECS integration), so we'd have to figure out how to deal with that

This is the gist - https://gist.github.com/maoo/f86eef1e99563a5cfaa33df5371b6804

Before moving forward with this, I'd consider the Helm option; it's reusable, production-ready, works across providers, and probably fairly easy to write, starting from the docker-compose we already have.

I guess we can make the Docker ECS integration work, but with some tweaking and the uncertainty that new docker cli versions will break it. Maybe there are also alternatives - that I couldn't find - on how to deploy Docker Compose on AWS.

Eager to hear your thoughts. Thanks!

@maoo
Copy link
Member

maoo commented Feb 5, 2024

Quick update on this. We decided to start spinning up an EC2 instance that runs docker-compose, while - in parallel - build support for cloud-native deployments.

We'll update this issue as soon as we have a running endpoint of TraderX on FINOS.

@maoo
Copy link
Member

maoo commented Feb 23, 2024

Hey folks, quick update; after some iterations, we now have https://demo.traderx.finos.org/ up and running.

We COULD probably close this issue, although we believe that in order to have a "live demo" we should probably do some basic UI/UX work.

@rocketstack-matt
Copy link
Member

Shall we track that work on a separate issue? Ultimately what we have now is a live view of what it is (for better or worse)

@maoo
Copy link
Member

maoo commented Feb 27, 2024

Shall we track that work on a separate issue? Ultimately what we have now is a live view of what it is (for better or worse)

Opened #125 , closing this one. Thanks everyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants