Having them two seems to create a conflict and as the result, databricks cannot execute anything in the notebook. My workaround is to uninstall azureml-sdk[databricks], restart cluster then install azureml-sdk[databricks] after Tensorflow is already in place. But I had to do this everytime my cluster restarts and it's annoying.
I use Databricks runtime 5.2