Issues related to dev branch .yml and some documentation

[RaSi96]

Hello team,

This is my first issue ever here on GitHub, so please let me know if anything is amiss. I am a bit apologetic that my problems have something to do with this fantastic software, but my questions have been mounting over the last few days. The source of my issue here is that superset, as of the time of my writing, still for some reason doesn't support simple scatterplots. (See this comment.)

1. Since bubble charts aren't a 1:1 replacement for a scatter for for me (they require 3 mandated variables?) as recommended here, I decided to use a line plot and arrived at PULL 17917. However, starting a non-dev superset instance using docker-compose -f docker-compose-non-dev.yml up and attempting to use a line chart unfortunately still asks me for a temporal dimension - is this because I am running the non-dev branch or is it something else? (See this comment.)

2. I then assumed that the dev branch would be a bit further ahead, so I updated my superset folder using git pull origin master and docker-compose -f docker-compose-non-dev.yml pull (both of which succeeded), but using docker-compose -f docker-compose.yml pull succeeds with everything except superset-websocket: WARNING: Some service image(s) must be built from source by running: docker compose build superset-websocket. pull access denied for superset-websocket, repository does not exist or may require 'docker login': denied. Running docker compose build superset-websocket succeeds, but retrying docker-compose -f docker-compose.yml pull returns the same error. (See this comment.)

3. [Edited after my third comment] Attempting to run the dev branch using docker-compose -f docker-compose.yml up struck me with ISSUE 7800. I followed these suggestions from there:

• changed the docker-compose.yml db.port from 5432:5432 to 5433:5432 and restarted,
• changed the docker-compose.yml db.port from 5432:5432 to 5433:5432, commented out the entire superset-tests-worker section and restarted,
• commented out the docker-compose.yml db.port line and restarted,
• commented out the docker-compose.yml db.port line and the superset-tests-worker section and restarted,
• tried using some of the settings from the non-dev.yml file and restarted.

But nothing worked. Each and every time superset services kept going into an endless error loop where one service would exit with error[1] and cause all the other images to fail with error[1], constantly printing out their error messages to my terminal until it froze:

• either my postgres instance can't be found at all (Are you sure the server is running at and listening on port 5432?), or I get the following messages:
• From superset_worker, superset_worker_beat, superset_app: (psycopg2.OperationalError) could not translate host name "db" to address: Name or service not known.
• From superset_db: FATAL: database files are incompatible with server superset_db; DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 14.2 (Debian 14.2-1.pgdg110+1).

4. do the dev and non-dev branches differentiate between development of the superset software or development of data visualisations? (See this comment.)

5. when I use openssl rand -base64 42 I get numbers and symbols; using this as my secret key doesn't work due to the symbols. Am I supposed to include them?

6. another warning that keeps popping up is the $CYPRESS_CONFIG variable is not set. Defaulting to a blank string. when using docker-compose -f docker-compose.yml up. Is this something I should be setting? I have searched in docker-compose.yml as well as superset_config.py, but haven't found anything that's the $CYPRESS_CONFIG. (See this comment.)

Additionally if I may, I feel like this is something that should really be addressed in the documentation for superset: running superset in a docker container means using localhost:5432 to connect to a postgres database won't work even if psql is listening on the applicable port because localhost, to the docker container, means it only sees itself, not the actual host machine. Instead, pg_hba.conf should be configured to listen on the container's IP and superset should be given the host machine's ipv4/ipv6:5432 to successfully connect (see this article which is also of great help.) These little details took me a rather surprising amount of time to digest :)

Once again I apologise for bringing what seems to be an exhaustive list of problems with superset, but any help with these issues would be tremendously appreciated. I'm quite certain it's something very elementary that I'm overlooking, because updating my superset copy and using the non-dev branch is smooth as butter; only the dev branch causes an issue. I came across this project a week ago and have since thoroughly enjoyed using it whenever I can, and I'm very excited to see this project succeed.

EDIT: I thoroughly forgot to add in system information.

• I am using the Arch Linux kernel: Linux 5.17.5-arch1-1
• I am running this on KDE Plasma: 5.24.4
• I use Superset through my Firefox browser: 99.0.1-1
• I am using PostgreSQL version: 14.2-1
• I am using docker version: 1:20.10.14-1
• I am using docker-compose version: 2.5.0-1
• I am using Python version: 3.10.4-1

Please let me know if any more information is required. In the interim it appears I'm resigned to the superset non-dev branch for most of my work. (See this comment.)

[RaSi96]

Small update: I have come across a workaround in ISSUE 16432 that addressed Q2 (which I have stricken.) The workaround was the first half of nytai's comment to comment out the superset-websocket service part of the docker-compose.yml file. This allowed me to then successfully run docker-compose -f docker-compose.yml pull.

Questions 1 and 3-6 have yet to be addressed.

EDIT: A bit more information, upon searching for the phrase in Q3.3 (FATAL: database files are incompatible with server superset_db), the only 2 relevant results I have come across are 1. over here and 2. over here. I would like to say that the only postgres database I have created and exposed specifically for superset was made just 2 days ago after a fresh install of postgresql version 14.2-1. Version 10.0-1 was released on 6-Nov-2017.

[RaSi96]

Another quick update (before I call it a night): following the paths within the docker-compose.yml file lead me to a readme.md under ~/path-to-superset/docker/ that, under the '## Production' section, uses language unambiguous enough to answer Q4 (which I have stricken): the dev branch is indeed for developing the superset software and non-dev is for production.

This also somewhat provides context for Q3: maybe the reason I am seeing a lot of errors is because I'm missing some kind of environmental setup that permits development, though I can't really be sure of this.

Questions 1, 3 and 5-6 have yet to be addressed.

[RaSi96]

Many updates:

• Q1: This issue was caused because I had not set the feature flag GENERIC_CHART_AXES to True. Apologies for misunderstanding this when reading through PULL 17917, however after reviewing the pull and cross-referencing the feature flag documentation it is apparent that there is a related problem: a lot of feature flags aren't actually documented. I have stricken this issue out.

• Q3: Given the feature flag functionality and the fact that (Q4) - the dev branch is for superset software development, there isn't necessarily a need right now for me to be using the dev branch at all. This implicitly takes care of Q6 as well, but I'll leave Q3 as it it's still a little unclear to me what the problem is there, and I hope to find a solution for it before the next time I need to launch the dev branch.

• (see Q3 just above this.)

Additionally, I am unsure who to thank for this but it would seem that turning GENERIC_CHART_AXES to True also now includes the option for a simple scatter plot, which was quite the pleasant surprise. A big thank you to the team.

It would appear the crux of my problem has been solved by the age old solution of RTFMing with a fine-toothed comb, though I guess the teeth of my comb could be a little larger if the documentation was ever-so-slightly more encompassing. Only Q3 and Q5 remain.

[jbat]

@RaSi96
After installing with docker compose successfully in early 2022, and now revisiting SuperSet in early 2023 I also have hit the loop issue with

superset_db DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 14.2

Seemed clear that the my previous install db data dir was being pointed to somewhere.
In docker-compose.yml the db service references the volume where the db data is stored.

volumes:
      - db_home:/var/lib/postgresql/data

Listing the volumes in docker

docker volume ls
DRIVER    VOLUME NAME
local     superset_db_home
local     superset_redis
local     superset_superset_home

Solution was to remove the old volume for db_home and rerun docker compose to init the database correctly.
( of course this blows away all data from the previous install )

docker volume rm superset_db_home
docker-compose -f docker-compose-non-dev.yml up`

[RaSi96]

Hi @jbat, thanks for responding!

Your solution seems to make sense, in response I decided to revisit Superset and give it a whirl again today, reinstalling from scratch using docker-compose, but (un)fortunately I wasn't affected by the loop issue I mentioned under (3), though I also was unable to connect to my PSQL instance using either the instructions in the documentation or the article I linked to here. Perhaps this issue is only caused once Superset is made aware of a database? Additionally, would enabling loading example datasets somehow contribute to this? I followed the installation instructions from the documentation and everything up to docker-compose -f docker-compose-non-dev.yml pull worked fine. Before running docker-compose -f docker-compose-non-dev.yml up however, I disabled loading of the example dataset and visuals and it started up like butter.

For reference, no part of my system has changed except for the versions of all my software. This time I tried with Arch 6.1.11, Postgres 15.1, Docker 1:23.0.1, Docker-Compose 2.16.0, Python 3.10.9.

EDIT: I missed a step when setting up my database connection but even with PSQL accessible, I still don't face the loop issue.

EDIT: Much later, I still don't seem to be facing this issue when using Superset nowadays, so verifying whether it's the pointer to the database or not is something I can't do, heh. Since the meat of this issue has been addressed, I'll close it.