Renderfarm submitters
WIP Sorry, there is no detailed tutorial or guide for this at the moment. Basically you need to set up one of the following Render Farm environments, and create a suitable submitter. If the referenced Render Farm Submitter is detected, Meshroom will show an additional button. Setting up the environment can be complicated, so it is nothing for the casual user. (You would need different servers (physical or virtual) that can distribute and manage the Job to connected machines (Example). There will be a tutorial or guide on this in the future, but for now you are on your own. (So please don´t open a new issue on this just for asking for help)
https://renderman.pixar.com/tractor https://renderman.pixar.com/forum/download.php https://renderman.pixar.com/store
meshroom/submitters/simpleFarmSubmitter.py meshroom/submitters/simpleFarmConfig.json
https://alicevision.org/img/meshroom/renderfarm.png
https://github.com/alicevision/meshroom/pull/81 https://materialsproject.github.io/fireworks
No MR submitter so far...
https://github.com/alicevision/meshroom/issues/1039 https://www.youtube.com/watch?v=OYVZvlXdBsg (reference) http://ramellij.blogspot.com/2015/06/how-to-build-open-source-renderfarm.html
[Experimental]
- Get the Meshroom OpenCue Submitter https://github.com/alicevision/meshroom/pull/992
- Place it in the Meshroom Submitters folder
- The Meshroom Submitter is using OpenCue PyOutline.
- Cuebot server with OpenCue database, RQD Render Agents on the target machines, PyCue and PyOutline on the primary machine are required to be set up. https://www.opencue.io/docs/concepts/opencue-overview/
- The process is described here: https://www.opencue.io/docs/getting-started/
- RQD can now run a submitted Meshroom Job on the render-farm machines
This can be setup in the Google Cloud environment as described here: https://cloud.google.com/solutions/creating-a-render-farm-on-gcp-using-opencue https://cloud.google.com/opencue
https://www.youtube.com/watch?v=uOi3azKJ3Xs (reference) https://www.youtube.com/watch?v=Vk-huejruG0 (reference for local setup)
Note: This has not been tested
https://github.com/alicevision/meshroom/issues/357#issuecomment-535851216
- slurm on High-Performance-Computing-Cluster / render farm
Slurm needs bash script to load jobs with specific comments such #SBATCH (https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html) .
slurmFarmSubmitter converts the Meshroom nodes to those scripts through python + jinja template https://github.com/sergiy-nazarenko/hafarm/blob/master/slurm_job.schema
slurmFarmSubmitter.py needs to be placed in meshroom/submitters/slurmFarmSubmitter.py
python subprocess module to submit a Slurm sbatch job:
https://github.com/alicevision/meshroom/blob/dev/slurm/meshroom/submitters/slurmSubmitter.py
Details on Slurm:
https://vsoch.github.io/lessons/slurm/ (good overview and tutorials)
https://aws.amazon.com/de/hpc/parallelcluster/
https://docs.aws.amazon.com/parallelcluster/latest/ug/schedulers.slurm.html
https://cloud.google.com/blog/products/gcp/easy-hpc-clusters-on-gcp-with-slurm
https://cloud.google.com/solutions/deploying-slurm-cluster-compute-engine
https://github.com/alicevision/meshroom/wiki/Meshroom-on-AWS
leverage AWS SQS (Simple Queue Service), S3 (Simple Storage Service) and boto https://github.com/boto/boto
to do: submitter
reference: https://www.youtube.com/watch?v=_Oqo383uviw
Untested. Starting point: https://www.jigsaw24.com/articles/use-backburner-to-queue-renders-from-non-autodesk-software
It means that an instance is planning to compute this node. For instance, if you open multiple Meshroom on the same files (usually from multiple machines sharing a storage). It is important when you submit a job on a renderfarm that will manage the scheduling on multiple machines. (https://github.com/alicevision/meshroom/issues/1165#issuecomment-733601561)
Other Render Managers to evaluate:
https://coalition.readthedocs.io/en/latest/
https://prism-pipeline.com/pandora/ (looks to be easy to set up) https://github.com/chadmv/plow