Multiple definition files

[fenos]

Hello, Awesome work with atlas!

I was wondering if you guys could consider the possibility of splitting the schema into multiple .atlas.hcl files and use a glob like syntax in the cli, or have a master file that we can import other files.

This would help organising the tables and definitions in a neater way

[masseelch]

Hey @fenos, thank you very much for your kind words and for reaching out to us.

I think that supporting to split the HCL definitions into multiple files is a good idea and that we should support it soon.

I'll discuss how to tackle this best and will provide to you a design proposal to evaluate with your use case. Is this okay with you? If you have some design ideas already, feel free to share them too 😊

[JoeyMckenzie]

+1 for this. Coming from Prisma, the same issue exists there and was solved with simple JS command tools to aggregate the partial schema files into the main schema.prisma file that it ultimate looks for. Been working on a side/learning project using atlas and wanted this same functionality, so I ended up implementing my own file merging process (just a simple Go program). Without too much context (sorry), the program simply looks for .hcl files in a root /atlas directory and copies them into the root atlas.hcl file.

This is obviously just for my needs within the project and does nothing in terms of schema validation/convention enforcement/etc. but would be nice to eventually have as a CLI option of some sort.

[JoeyMckenzie]

Following up on this. Currently working on this in my fork as I'd love for this functionality to be included in the CLI. Design ideas as follows:

• Users can define partial definition files with some sort of conventionalized file extension (.atlas.hcl) so we know it's specifically an HCL file atlas can understand (my_partial_schema_def.atlas.hcl)
• Offer options to output the file with a specific name
• Two approaches on how to organize partial files:
  • Store all partial files in a directory near the root of your project that should be output in an aggregate atlas.hcl file at the root
  • Co-locate .atlas.hcl files with their specific domain (my preference), allows for better modularity and storing only relevant schema definitions within their particular concern/feature

Without some sort of glob importing syntax specific to atlas, the process would involve finding all .atlas.hcl files in the project and doing a file merge with the output being the main atlas.hcl file. I think both options above would be great to offer up to users. First option is the more simple case, just locating a directory and combining all schema files into an aggregate. The second requires a file walk, and I think it would nice to offer some sort of directory ignoring to avoid walking the file system in places where there would be no atlas files (like a project containing a frontend with node_modules). All food for thought, I'm open for any suggestions.

With some sort of importing syntax in the parent file, would this be a specific implementation to .atlas.hcl files? I'm relatively new to Go and can't seem to find (read: didn't Google hard enough) a way to extend .hcl files natively, so using imports seems like it would be an extension specific to atlas.

[rotemtam]

Hey @JoeyMckenzie !

Thanks for contributing to this discussion. We are touching here on decisions that would have immense impact on the project - basically the modularization mechanism for the DDL. We need to factor many things here, including our future plans of having projects deal with multiple databases in a single data topology. In addition we are working on a versioned migrations workflow (similar to "traditional" migration tools) and that would influence project structure as well. Im aiming for a simple, elegant solution which from my experience will take time to arrive at.

Therefore while I understand the need, I prefer to delay an official and final decision on the DDL level until the project evolves more and rely on the community to produce tools like you mentioned to bridge the gaps in the meantime.

I hope this makes sense

[masseelch]

With the recent release of Atlas configuration files this issue can be closed. if you think it is not, feel free to re-open this.