New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Standalone deb builds #1701
WIP: Standalone deb builds #1701
Conversation
|
To finish this I would need to know what are the actual build dependencies of ColumnStore (https://jira.mariadb.org/browse/MCOL-4471) and what are the required end products? Would just one mariadb-plugin-columnstore.deb and .rpm suffice? Or do you also need to produce other parts of the server? |
|
Currently the Drone CI spits out these extra packages in addition to the ColumnStore binary package: Is it really necessary to build all these? Would it not suffice to just build the ColumnStore package and then pull in the necessary MariaDB Server bits from the corresponding mariadb.org repository are prebuilt binary packages? |
because of the package versions conflicts. packages from develop branch built against HEAD of 10.6 mariadb server repo. mariadb server prebuilt binary packages would have conflicts most of the time. |
Can you please elaborate on this? If the issue is simply package metadata in .deb, we can fix it easily. I am interested here in what are the actual functional dependencies/relations? |
you can try it by yourself. just add official mariadb repositories and try to install columnstore packages from some of recent develop branch builds. and yes, this is simply package dependencies problem (metadata in .deb). so if you can fix it easly – you are welcome! |
That was not the point. I am asking you what the dependency is? I am quite sure putting any The metadata in the deb exists only to reflect what the dependency is. Now you are telling me that there is no dependency and the version strings in the .deb packaging are artificial. Surely there is a real dependency and you as ColumnStore developers should know it. |
I talk in context of you initial comment about 'extra packages'. Users usually get packages, not the ha_columnstore.so file. We have no independent build system and build process (from mariadb server project). We just build server with columnstore plugin. Artefacts of the build are packages. So, feel free to rework all the build process (from binaries to packages)! As for real dependency from server I know nothing about that. As far I do only CI part. I think plugin interface should be compatible across major (or minor?) versions. Let's wait for answers from 'real' developers like @drrtuy about that. |
That was my original idea but the actual implementors of the rubuilding process prefered to go the other way. |
|
Maybe that route was taken due to the lack of documentation of *actual*
dependencies?
If you publish a list somewhere (e.g.
https://jira.mariadb.org/browse/MCOL-4471) then I could engineer the
build + Debian packaging to produce only the ColumnStore itself and omit
e.g. s3 plugin and other cruft that is surely not needed.
|
|
7bbf4b6
to
dbfe055
Compare
|
I believe we had merged the idea with another patch of yours so closing this one. Plz msg me if I am wrong @ottok . |
|
This is a separate idea/PR, please don't close it.
|
dbfe055
to
f15da92
Compare
|
@drrtuy How do I trigger the CI for this PR? None seem to have run now after I force pushed an improved version of this. |
51de141
to
8545012
Compare
|
The general structure is correct, but builds fail on some pthread issue maybe due to missing dependency? https://ci.columnstore.mariadb.net/mariadb-corporation/mariadb-columnstore-engine/2043/7/4
|
8545012
to
53f6bfc
Compare
|
@drrtuy This might be the root cause: Where is the API version defined/compared? On the server side seems to be Is it in There is no material dependency to the server, you simply define that you always depend on the same server as built with? |
877dfc4
to
9ae4434
Compare
|
Seems the ColumnStore build itself is very slow, and trimming away the other parts don't help much in build speed. However, using ccache does speed up the build 90%. Tried integrating drone-cache but can't complete it without your help, as it would need a new S3 bucket to store cache in: |
|
After hacking the server version scheme ColumnStore does install, but write operations die: I was not able to debug why. |
I will run the build and look at the server trace when MDB thread crashes. I will message you back with the results. |
It takes a way more cpu cycles to compile MCS for sure than the server itself. Let's see if drone cache could reduce the timings. |
…dependency for debs" This reverts commit 95959f8.
Building the entire MariaDB Server and all the binaries and packages it includes is a waste of resources. A better approach is to build only the minimal core server and ColumnStore, and nothing excess. This brings many benefits: - The build is faster, yielding shorter turnaround time for devs - The build is faster, making the CI run faster and saving both CPU, disk space, time and money. - If "upstream" MariaDB Server has random changes in their debian/* directory, the stand-alone ColumnStore builds are most of the time unaffected and CI will not break. Upstream improvements can by synced to ColumnStore in the chosen extent and at a suitable time. - Decoupling like this makes the code base cleaner and easier to maintain. For example the real ColumnStore build dependencies will be much more obvious in the future with this structure.
level=error name=drone-cache ts=2021-05-23T06:42:11.024759994Z caller=main.go:466 err="[WARNING] build cache, process rebuild failed, mount <.ccache>, make sure file or directory exists and readable stat .ccache: no such file or directory\n"
604b32b
to
b04e5ca
Compare
Build only ColumnStore and produce only the mariadb-plugin-columnstore binary package to be able to publish it independently of the server.
DO NOT MERGE THIS. This is just an experiment created as a PR to get the Drone CI run on it.