[META] Use automated testing / Add .deb/.rpm/pacman packages

[ehfd]

• Build and add .deb and .rpm packages (help for Arch or Alpine distros also welcome)

We should definitely do this, but we need help. This requires build procedures for every relevant distro.

• Testing workflows

In every PR, we should test using Selenium and Firefox an end-to-end procedure, starting from container builds to testing that GStreamer starts, and we see that the Start button of the web interface is visible.

• Formatting, automatic code assessment tools, and a single syntax for code documentation

We should incorporate format checking such as pylint or yapf and code testing. Similarly for JavaScript. Code checking and assessments should automatically be done when a Pull Request starts.
People with relevant experience are welcome to reach out.

Code documentation should also be added to the code itself. There are well-documented locations and those that aren't really so.

These should be done before the codebase becomes substantially larger than now.

[ehfd]

https://github.com/pacstall/pacstall
I also stumbled upon this; an AUR for Ubuntu?

[ehfd]

@danisla
The following are outdated actions which require major updates to the pipeline:
tj-actions/changed-files@v1.1.2 (newest version does not work just by changing the tag)
actions/create-release@v1 (archived and deprecated)
actions/upload-release-asset@v1 (archived and deprecated)

[ehfd]

@danisla The Actions pipeline is buggy. The release for GStreamer is always botched for some reason when Build & publish all images is not run before Publish release, and the actions are outdated.

Ref: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

[ehfd]

@danisla It seems that the release pipeline is broken. GStreamer is not delivered correctly especially in Ubuntu 20.04 and apparently breaks something with the caching mechanism.

For now, upload releases manually.

[danisla]

@danisla It seems that the release pipeline is broken. GStreamer is not delivered correctly especially in Ubuntu 20.04 and apparently breaks something with the caching mechanism.

For now, upload releases manually.

The pipeline fixes should be a separate bug, adding .deb packages is a new feature.

[ehfd]

@danisla I won't open a new issue because I feel the CI is nearly fixed, but is it intended that gst-py-examples containers are not built in "Build & publish changed images"?

[danisla]

I was attempting to only build the gst-py-examples if any of the matrix tasks changed, but it's not obvious that this is possible ATM.
This commit should always build the example images: 587ec41

[danisla]

@ehfd for new releases, the process has changed slightly. The pipeline no longer creates the GH release.

1. manually create the Github Release and Tag.
2. Generate the release notes with the github button.
3. The pipeline will trigger once the release is published and will upload the assets to the release.

[ehfd]

Default branch was changed to main to comply with GitHub guidances. If you have a local clone, please use:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

[ehfd]

@ehfd for new releases, the process has changed slightly. The pipeline no longer creates the GH release.

1. manually create the Github Release and Tag.
2. Generate the release notes with the github button.
3. The pipeline will trigger once the release is published and will upload the assets to the release.

I have complied to this process. Best way is to set the new release as a pre-release until the assets are uploaded then change to latest.

[danisla]

Default branch was changed to main to comply with GitHub guidances. If you have a local clone, please use:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Are you going to update all the things that depend on master?

[ehfd]

Default branch was changed to main to comply with GitHub guidances. If you have a local clone, please use:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Are you going to update all the things that depend on master?

I am doing it.

[ehfd]

I am doing it.

Done.

I am freezing both the desktop containers and Selkies-GStreamer after v1.4.3 and withdrawing for a few months, unless immediate issues arise.

Please ask me before a release, as it can affect the desktop containers.

[danisla]

Not sure if I’m ok freezing all Selkies gstreamer releases for a few months. Can you lock the desktop containers to Selkies-gstreamer v1.4.3? Having them roll with latest doesn’t seem like good CI practice anyway.

[ehfd]

Not sure if I’m ok freezing all Selkies gstreamer releases for a few months. Can you lock the desktop containers to Selkies-gstreamer v1.4.3? Having them roll with latest doesn’t seem like good CI practice anyway.

If so, you can make a release, no problem.
But I still do want to roll with latest.
I'll manually trigger a rebuild when I can check things. That way the container images will be alive even after releases introduce incompatibilities.

[ehfd]

Unless something is very out of the ordinary, the container desktops will be kept as-is for at least a few months (only Selkies-GStreamer dependencies will be updated as there are new releases).
I have exhausted and fixed every reason the container might break.

@danisla You are free to make releases, just contact and tell me when you have changed the GStreamer Dockerfile (or its runtime dependencies in Dockerfile.example). Anything otherwise will not interfere with the containers.

[ehfd]

@danisla
It seems a good idea to:

1. Create a merged .deb package which includes the Python, GStreamer, Web, and the Interposer.
2. Ship a .tgz archive of the interposer .so file, because one of the advantages of using Selkies is not having to install as root.
3. How would it be possible to create a completely portable version of all the components?

[ehfd]

@danisla build_changed_images is broken, looks like it's incomplete.

selkies-gstreamer/.github/workflows/build_changed_images.yaml

Lines 98 to 133 in 2d69739

	# Note: When modifying this job, copy modifications to all other workflows' image jobs.
	changed_example_images:
	needs: changed_component_images
	runs-on: ubuntu-latest
	strategy:
	matrix:
	include:
	- name: gst-py-example
	version_suffix: -ubuntu20.04
	push_image: "false"
	build_args: PACKAGE_VERSION=0.0.0.dev0;UBUNTU_RELEASE=20.04;GSTREAMER_BASE_IMAGE_RELEASE=pr${{ matrix.version_suffix }};PY_BUILD_IMAGE=ghcr.io/selkies-project/selkies-gstreamer/py-build:pr${{ matrix.version_suffix }};WEB_IMAGE=ghcr.io/selkies-project/selkies-gstreamer/gst-web:pr${{ matrix.version_suffix }};JS_BASE_IMAGE_RELEASE=pr${{ matrix.version_suffix }};JS_BASE_IMAGE=ghcr.io/selkies-project/selkies-gstreamer/js-interposer
	dockerfile: Dockerfile.example
	source_directory: .
	source_files_for_diff: |
	addons/**
	src/**
	Dockerfile
	Dockerfile.example
	pyproject.toml
	setup.cfg
	.github/workflows/build_and_publish_changed_images.yaml
	- name: gst-py-example
	version_suffix: -ubuntu22.04
	push_image: "false"
	build_args: PACKAGE_VERSION=0.0.0.dev0;UBUNTU_RELEASE=22.04;GSTREAMER_BASE_IMAGE_RELEASE=pr${{ matrix.version_suffix }};PY_BUILD_IMAGE=ghcr.io/selkies-project/selkies-gstreamer/py-build:pr${{ matrix.version_suffix }};WEB_IMAGE=ghcr.io/selkies-project/selkies-gstreamer/gst-web:pr${{ matrix.version_suffix }};JS_BASE_IMAGE_RELEASE=pr${{ matrix.version_suffix }};JS_BASE_IMAGE=ghcr.io/selkies-project/selkies-gstreamer/js-interposer
	dockerfile: Dockerfile.example
	source_directory: .
	source_files_for_diff: |
	addons/**
	src/**
	Dockerfile
	Dockerfile.example
	pyproject.toml
	setup.cfg
	.github/workflows/build_and_publish_changed_images.yaml

[ehfd]

TODO: Change https://github.com/selkies-project/selkies-gstreamer/blob/main/.github/actions/build_and_publish_image/action.yaml with https://github.com/docker/build-push-action.

[ehfd]

Remaining tasks:
Build and add .deb and .rpm packages (help for Arch or Alpine distros also welcome): This is still preferred despite considering Conda and AppImages.
Formatting, automatic code assessment tools, and a single syntax for code documentation: Consider SonarCube as a potential possibility.