Dynamically generate valid xcode project files from json #9

BPerlakiH · 2023-12-15T08:48:38Z

Fixes #1
Fixes kiwix/kiwix-apple#603
Fixes kiwix/kiwix-apple#559
Fixes #10
Fixes kiwix/kiwix-apple#615

How it works

checkout the main Kiwix apple branch, from kiwix/apple (currently it is using a dedicated branch, but once that's merged we can use main).
checkout this repository
run the built in scripts to generate: info plist files, config files, and finally generate a custom project file that is re-using all the settings from the kiwix/apple main branch, but adding custom targets:

next steps

the next step is to build the apps from the generated project using fastlane.

other changes

it is renaming the DWDS folder to be lowercase: dwds
it is adding python scripts to do most of the parsing / transforming / downloading steps, with unit tests

…nfig value

This reverts commit 5ef6d7f.

This reverts commit e30a8a1.

BPerlakiH · 2024-01-11T12:52:42Z

Thank you!

I think we lack explanations regarding how this technically works: I might have missed something but it looks like any push to main or PR will trigger a build and testflight beta for all custom apps. Is that what we want?

There doesn't seem to be a way to (even manually) trigger a single app build as well. It even looks like it's done step by step for all apps so a broken app would fail the whole thing. Having a single-app tool would allow to run it in parallel in github actions and even have a per-app reporting in the workflow.

On the python code, I have some general comments

generate_and_download.py: I don't see the value of this file at the moment. If it's entirely dependent on custom_apps.py and is entirely static code, maybe it should be move in __main__ of custom_apps.py

you should use pathlib module for filepath handling and not rely on strings. This will be handy for stuff like rglob

There's no dosctring and naming is not helping: CustomApps().append_to(): no clue from naming what this is supposed to do.

the chaining of string replacements eventually ending up being executed is terrifying 😅 This is extremely opaque and hard to maintain.

use f-strings instead of .format() ; it's clearer and in line with our convention.

split the Info holder and the command builder.

dosctring or comments for all commands

fix codefactor issues

use named groups for your regexp

there's a TODO:remove me comment. is it still necessary?

don't us the shell to perform simple operations for which there's an API (file copy for instance)

use subprocess (with arrays) when running processes

I have fixed these issues, please review.
Next step is to look into the CI / CD separation, as mentioned above.

…script

kelson42 · 2024-01-12T08:19:57Z

@BPerlakiH CI still fails!

rgaudin

Thank you ; it's better but I think the priority should go into making it per-app first and then improve the code quality.

rgaudin · 2024-01-12T09:52:28Z

src/custom_apps.py

-    def append_to(custom_plist="Custom.plist"):
-        for cmd in InfoParser.plist_commands():
-            os.system("{} {}".format(cmd, custom_plist))
+    def create_custom_project_file(self, path=Path()/'custom_project.yml'):


You should not set it here as this is set once at import time. Better to leave it None and in the function set it if it's None

rgaudin · 2024-01-12T09:53:51Z

src/generate_and_download.py

-    # create the xcconfig files
-    custom_apps.create_xcconfigs()
+    # create the plist files
+    custom_apps.create_plists(custom_plist=Path()/"Custom.plist")


Suggested change

custom_apps.create_plists(custom_plist=Path()/"Custom.plist")

custom_apps.create_plists(custom_plist=Path("Custom.plist"))

rgaudin · 2024-01-12T09:55:29Z

src/info_parser.py

+            out_path = self._info_plist_path()
+            # create dir, if doesn't exists yet
+            if not out_path.parent.exists():
+                out_path.parent.mkdir()


Suggested change

out_path.parent.mkdir()

out_path.parent.mkdir(parents=True, exists_ok=True)

then you dont need the condition

rgaudin · 2024-01-12T09:56:03Z

src/info_parser.py

+
+    def create_plist(self, based_on_plist_file):
+        with based_on_plist_file.open(mode="rb") as file:
+            plist = plistlib.load(file)


Suggested change

plist = plistlib.load(file)

plist = plistlib.load(based_on_plist_file.read_bytes())

rgaudin · 2024-01-12T09:57:20Z

src/info_parser.py

-
+
+    def _info_plist_path(self):
+        return Path()/self.brand_name/f"{self.brand_name}.plist"


Suggested change

return Path()/self.brand_name/f"{self.brand_name}.plist"

return Path(self.brand_name) / f"{self.brand_name}.plist"

kelson42 · 2024-01-12T08:32:52Z

.github/workflows/ci.yml

+          |
+          cd apple 
+          bundle exec fastlane ios beta 
+          # bundle exec fastlane mac beta


Not sure why all these secrets are needed to just build the app and not upload it!

Why is macOS not tested?

kelson42 · 2024-01-12T08:33:55Z

CONTRIBUTION.md

+
+## At the beginning we have the following:
+- custom brand folders, each with an info.json file, and xcassets
+- we have the kiwix/apple code (as a seperate repository). 


kiwix/apple1 or link.

kelson42 · 2024-01-12T08:34:12Z

CONTRIBUTION.md

+
+## What we need in order to build a custom app?
+
+### A Zim file


kelson42 · 2024-01-12T08:34:29Z

CONTRIBUTION.md

+This needs to be downloaded from the url given in the `info.json`, and placed under the branded folder, eg: `/custom/dwds`
+
+### A .plist file
+The plist file is a list of settings, that is used by xcode for the given target to be built. It is in a required xml format.


kelson42 · 2024-01-12T08:35:38Z

CONTRIBUTION.md

+### A .plist file
+The plist file is a list of settings, that is used by xcode for the given target to be built. It is in a required xml format.
+
+It is created under the folder with the same name, eg.: `dwds/dwds.plist`,


This file is automatically generated and should not be edit manually. This should be clear.

kelson42 · 2024-01-12T08:36:36Z

CONTRIBUTION.md

+
+# A new custom_project.yml file
+The main `kiwix/apple` repo contains a `project.yml` file, it describes all the common build and project settings, and all the templates we re-use to create a final xcode project file, which is used to build the custom apps. At the end each custom app will be a separate target we can build.
+Therefore we need dynamically create the `custom_project.yml` file, which will import the `project.yml`, and sets up the new targets (one for each brand), it looks more or less, like this:


we dynamically create

kelson42 · 2024-01-12T08:39:14Z

src/custom_apps.py

@@ -0,0 +1,59 @@
+from pathlib import Path


The code should be for only one specific custom app. If we want to make all custom apps at the same time, this will be done at a higher level.

kelson42 · 2024-01-12T08:42:12Z

tests/test.json

@@ -0,0 +1,13 @@
+{


Please make a fake app (not like a real one with real values). We should also avoid to download such a big file. I have prepared here a small test ZIM file https://dev.kiwix.org/apple/custom/test/wikipedia_en_ray_charles_maxi_2023-12.zim

kelson42 · 2024-01-12T10:03:30Z

CONTRIBUTION.md

@@ -0,0 +1,80 @@
+# Technical documentation for contributors


Usually this is called CONTRIBUTING.md I believe.

BPerlakiH · 2024-01-15T00:20:55Z

The required changes were merged in other PRs.

BPerlakiH added 7 commits December 15, 2023 09:47

Update build script

283e831

Move project file under apple folder

cc85271

Include enforced language

4aa5374

Update files

8c0014f

Add python parser

3ba2a22

Update parsing

44d99a2

Update script

3910a50

BPerlakiH marked this pull request as draft December 18, 2023 03:29

BPerlakiH added 22 commits December 29, 2023 20:48

Move out enforced langauge from xcconfig

68d08ae

Ignore DS_Store

cd68c42

Update python files, finding info files, and get enforced_language co…

8afabc8

…nfig value

Add brew dependencies

44a30e5

Update zim url for dwds

3b8fd9d

Update build file

43d5ab3

Install brew dependencies on CI

ec008c8

Checkout the current branch

07b2656

Update project file

75a80d3

Custom project generation, and spliting the project yaml

96c9e36

Add tests for exclude lang

acd1d9f

Update python project creation

847ee29

Ignore DS_Store

fdd5fe1

Update project creation, add Launch storyboard

39b725d

Add main generate file

1416951

Delete old configure file

496989f

Remove old info file

7e319de

Update the build steps to use python

bb1b868

Generate the custom project file from python

0019592

Fix project settings

4a2c079

Include all vars in info plist

f2ef1fa

Update workflow

f6a55f6

BPerlakiH added 6 commits January 11, 2024 00:35

Use named capture groups

ac408ad

Use pathlib

5ef6d7f

Revert "Use pathlib"

e30a8a1

This reverts commit 5ef6d7f.

Revert "Revert "Use pathlib""

499fb9a

This reverts commit e30a8a1.

Fix finding files several times

cfd7327

Update docs

f77b798

BPerlakiH requested review from rgaudin and kelson42 January 11, 2024 12:51

BPerlakiH added 8 commits January 11, 2024 23:53

Use quotes around the key

3a26c79

Tag validation for CD

9bb0c92

Merge remote-tracking branch 'origin/main' into feature/1-DWDS-build-…

bad2334

…script

Resolve merge

eec2496

Merge remote-tracking branch 'origin/main' into feature/1-DWDS-build-…

eaa74a0

…script

Merge remote-tracking branch 'origin/main' into feature/1-DWDS-build-…

2e97711

…script

Use multiline commands

9142a2e

Add checkout certs step

9ff2893

BPerlakiH added 2 commits January 12, 2024 10:26

Update secret keys

f10668d

Update CI step

53128a9

rgaudin requested changes Jan 12, 2024

View reviewed changes

kelson42 requested changes Jan 12, 2024

View reviewed changes

BPerlakiH added 5 commits January 12, 2024 15:00

Move tag validator, introduce Brand class

ecb6177

Run CD and build based on tag

c19fb58

Update args

e7c2875

Fix

0204f37

Fix mkdir

60c026b

BPerlakiH closed this Jan 15, 2024

BPerlakiH deleted the feature/1-DWDS-build-script branch February 14, 2024 21:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamically generate valid xcode project files from json #9

Dynamically generate valid xcode project files from json #9

BPerlakiH commented Dec 15, 2023 •

edited

Loading

BPerlakiH commented Jan 11, 2024

kelson42 commented Jan 12, 2024

rgaudin left a comment

rgaudin Jan 12, 2024

rgaudin Jan 12, 2024

rgaudin Jan 12, 2024

rgaudin Jan 12, 2024

rgaudin Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

kelson42 Jan 12, 2024

BPerlakiH commented Jan 15, 2024

	custom_apps.create_plists(custom_plist=Path()/"Custom.plist")
	custom_apps.create_plists(custom_plist=Path("Custom.plist"))

	out_path.parent.mkdir()
	out_path.parent.mkdir(parents=True, exists_ok=True)

	plist = plistlib.load(file)
	plist = plistlib.load(based_on_plist_file.read_bytes())



		def _info_plist_path(self):
		return Path()/self.brand_name/f"{self.brand_name}.plist"

	return Path()/self.brand_name/f"{self.brand_name}.plist"
	return Path(self.brand_name) / f"{self.brand_name}.plist"


		## What we need in order to build a custom app?

		### A Zim file

Dynamically generate valid xcode project files from json #9

Dynamically generate valid xcode project files from json #9

Conversation

BPerlakiH commented Dec 15, 2023 • edited Loading

How it works

next steps

other changes

BPerlakiH commented Jan 11, 2024

kelson42 commented Jan 12, 2024

rgaudin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BPerlakiH commented Jan 15, 2024

BPerlakiH commented Dec 15, 2023 •

edited

Loading