[DONE] improve clean_video_files and code cleaning #782

Badatos · 2023-03-21T11:06:00Z

Now clean_video_files command can clean obsolete video files AND User folders.

add --type and --dry options to clean_video_files
Replace python 3.7 by 3.10 in CI, as 3.7 EOL is 27 Jun 2023
Upgrade ldap3 package for python 3.10 compatibility
Accessibility testing are run with coverage
Code formatting: Remove useless " " in python strings

…rder

- Suppression des données propres à l'Université de Lille - Modification de la documentation

- Modification de la documentation

- Modification du ficher dockerfile-dev-with-volumes/config/elasticsearch/settings_local-7-exemple.py

- Modification de la documentation

…tion comments, add webp for podfile image extension and remove video extension for podfile document

…ation_markdown [DONE] Ptitloup/configuration markdown

…o feature_rmjquery

… to create it

…configuration Ptitloup/improve configuration

… root folder - create empty config

…configuration move configuration json file to main app and create md config file to…

…configuration remove unused commit in code formatting action

gitguardian · 2023-03-21T11:07:12Z

⚠️ GitGuardian has uncovered 11 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
-	Generic High Entropy Secret	`b3622f8`	pod/main/test_settings.py	View secret
5773735	Django Secret Key	`a72942b`	dockerfile-dev-with-volumes/config/elasticsearch/settings_local-7-exemple.py	View secret
5671468	Generic High Entropy Secret	`8b71cfd`	pod/main/test_settings.py	View secret
5671468	Generic High Entropy Secret	`bc9099a`	pod/main/test_settings.py	View secret
5671468	Generic High Entropy Secret	`4f0d61d`	pod/main/test_settings.py	View secret
5671468	Generic High Entropy Secret	`c0885a4`	pod/main/test_settings.py	View secret
5671469	Generic High Entropy Secret	`97ebbc8`	pod/authentication/tests/test_populated.py	View secret
5671469	Generic High Entropy Secret	`4233e7d`	pod/authentication/tests/test_populated.py	View secret
5671469	Generic High Entropy Secret	`d2ef31c`	pod/authentication/tests/test_populated.py	View secret
5671468	Generic High Entropy Secret	`d2ef31c`	pod/main/test_settings.py	View secret
5671468	Generic High Entropy Secret	`5241488`	pod/main/test_settings.py	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

ptitloup

Merci pour cette contribution

.github/workflows/pod.yml

.gitignore

pod/main/configuration.json

pod/video/management/commands/clean_video_files.py

ptitloup

ok pour moi

ptitloup · 2023-03-21T16:48:33Z

ok niveau code mais les tests unitaires ne passent pas, tu peux regarder pourquoi ? merci

Badatos · 2023-03-22T09:04:48Z

ok niveau code mais les tests unitaires ne passent pas, tu peux regarder pourquoi ? merci

Ca ne passe plus depuis le merge d'une des dernière PR sur les live :

FAIL: test_attributs (pod.live.tests.test_models.BroadcasterTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/Esup-Pod/Esup-Pod/pod/live/tests/test_models.py", line 139, in test_attributs
    self.assertEqual(broadcaster.transcription_file.url, "/media/testfile.vtt")
AssertionError: '/media/live_transcripts/broadcaster1.vtt' != '/media/testfile.vtt'
- /media/live_transcripts/broadcaster1.vtt
+ /media/testfile.vtt

Badatos · 2023-03-22T09:20:10Z

J'ai remplacé "testfile.vtt" par "broadcaster1.vtt" pour que ca passe les tests, mais pour faire bien je pense qu'il faudra ajouter un tests supplémentaire pour vérifier que testfile.vtt est bien présent quelque-part. (ou alors c'est qu'il ne faut plus préciser le fichier vtt à la création du Broadcaster)

pod/live/tests/test_models.py

dans les tests unitaire, tu as utilisé MEDIA_ROOT au lieu de MEDIA_URL

ptitloup · 2023-03-22T13:12:41Z

AssertionError: '/media/live_transcripts/broadcaster1.vtt' != '/home/runner/work/Esup-Pod/Esup-Pod/pod/me[33 chars].vtt'

/media/live_transcripts/broadcaster1.vtt

/home/runner/work/Esup-Pod/Esup-Pod/pod/media/live_transcripts/broadcaster1.vtt

ptitloup

ok pour moi, merci !

…%s_%s.png" % (video_to_encode.slug, thumbnail_path),` in `store_json_list_thumbnail_files`

LoicBonavent

La commande python manage.py clean_video_files supprime bien les fichiers inutiles.

ptitloup

ok

pod/video/Encoding_video_model.py

* 🐛 podfile fix pagination display (#787) * improve clean_video_files and code cleaning (#782) * Fix wrong url of paginated sub-themes (#789) * Add "help" command to makefile (#792) * Ptitloup/fix thumbnail (#790) * Hide channel desc (#794) * fix podfile (#798) * Correct podfile sidebar (#799) * Ptitloup/fix link playlist (#801) * Fix video theme (#800) --------- Co-authored-by: MattBild <34771705+mattbild@users.noreply.github.com> Co-authored-by: Olivier Bado-Faustin <bado@unice.fr> Co-authored-by: pampletousse <51699553+pampletousse@users.noreply.github.com>

* 🐛 podfile fix pagination display (EsupPortail#787) * improve clean_video_files and code cleaning (EsupPortail#782) * Fix wrong url of paginated sub-themes (EsupPortail#789) * Add "help" command to makefile (EsupPortail#792) * Ptitloup/fix thumbnail (EsupPortail#790) * Hide channel desc (EsupPortail#794) * fix podfile (EsupPortail#798) * Correct podfile sidebar (EsupPortail#799) * Ptitloup/fix link playlist (EsupPortail#801) * Fix video theme (EsupPortail#800) --------- Co-authored-by: MattBild <34771705+mattbild@users.noreply.github.com> Co-authored-by: Olivier Bado-Faustin <bado@unice.fr> Co-authored-by: pampletousse <51699553+pampletousse@users.noreply.github.com>

Mateo6051 and others added 30 commits January 17, 2023 12:34

get text for video errors

fc945c4

create command to export json configuration to markdown file

84b56ae

update configuration and improve export it to markdown file

bc5a543

add pydoc + gettexts

7550577

fix flake8 in createconfiguration

658e9d2

getttext removed error variable

cf8a7b3

remove unused markdown file, move some cofiguration from main to reco…

fd43999

…rder

show alert doouble removed

a013e0e

chapter added gettext

25d59f6

fix test for channel view when modify settings

ec9a2bb

fix recorder test views

12a551f

feature/conteneurisation :

a72942b

- Suppression des données propres à l'Université de Lille - Modification de la documentation

feature/conteneurisation :

40dc305

- Modification de la documentation

feature/conteneurisation :

75c419d

- Modification du ficher dockerfile-dev-with-volumes/config/elasticsearch/settings_local-7-exemple.py

feature/conteneurisation :

947fc15

- Modification de la documentation

feature/conteneurisation :

c8e5f98

- Modification de la documentation

add coding convention for global variables, remove space in documenta…

fe4cbab

…tion comments, add webp for podfile image extension and remove video extension for podfile document

remove unused override settings, import template settings

20e1b57

Merge pull request EsupPortail#710 from EsupPortail/ptitloup/configur…

9002301

…ation_markdown [DONE] Ptitloup/configuration markdown

Fixup: format code with Black

5b4741e

Merge branch 'EsupPortail:develop' into develop

0c9cfae

removed print url

cde99b5

Merge branch 'develop' of https://github.com/EsupPortail/Esup-Pod int…

2e66070

…o feature_rmjquery

unused imports removed

c01a885

add space and quote before title for markdown generation - use action…

1df1905

… to create it

Merge pull request EsupPortail#732 from EsupPortail/ptitloup/improve_…

26ae2a6

…configuration Ptitloup/improve configuration

move configuration json file to main app and create md config file to…

39ec95c

… root folder - create empty config

Merge pull request EsupPortail#733 from EsupPortail/ptitloup/improve_…

f387c78

…configuration move configuration json file to main app and create md config file to…

remove unused commit in code formatting action

f03549c

Merge pull request EsupPortail#734 from EsupPortail/ptitloup/improve_…

f1ad1eb

…configuration remove unused commit in code formatting action

Badatos changed the title ~~[WIP] Feature/improve archiving~~ [DONE] improve clean_video_files and code cleaning Mar 21, 2023

ptitloup requested changes Mar 21, 2023

View reviewed changes

Correct typo in configuration.json

744715e

ptitloup approved these changes Mar 21, 2023

View reviewed changes

Replace testfile.vtt by broadcaster1.vtt in pod/live/tests

18e3230

add "live_transcripts" folder in test_attributs (pod.live.tests)

667ce4f

ptitloup requested changes Mar 22, 2023

View reviewed changes

pod/live/tests/test_models.py Outdated Show resolved Hide resolved

Use MEDIA_ROOT and LIVE_TRANSCRIPTIONS_FOLDER in pod/live/tests

92ea2e5

ptitloup previously approved these changes Mar 22, 2023

View reviewed changes

ptitloup self-requested a review March 22, 2023 13:11

replace MEDIA_ROOT by MEDIA_URL

37ec0c3

ptitloup previously approved these changes Mar 22, 2023

View reviewed changes

Replace "%d_%s.png" % (video_to_encode.slug, thumbnail_path), by `"…

4736226

…%s_%s.png" % (video_to_encode.slug, thumbnail_path),` in `store_json_list_thumbnail_files`

LoicBonavent assigned LoicBonavent and unassigned LoicBonavent Mar 23, 2023

LoicBonavent self-requested a review March 23, 2023 09:36

LoicBonavent approved these changes Mar 23, 2023

View reviewed changes

ptitloup dismissed their stale review via 4736226 March 23, 2023 10:09

ptitloup approved these changes Mar 23, 2023

View reviewed changes

pod/video/Encoding_video_model.py Show resolved Hide resolved

ptitloup merged commit 2d594aa into EsupPortail:develop Mar 23, 2023

Badatos deleted the feature/improve_archiving branch March 23, 2023 11:01

Badatos self-assigned this Dec 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DONE] improve clean_video_files and code cleaning #782

[DONE] improve clean_video_files and code cleaning #782

Badatos commented Mar 21, 2023 •

edited

Loading

gitguardian bot commented Mar 21, 2023 •

edited

Loading

ptitloup left a comment

ptitloup left a comment

ptitloup commented Mar 21, 2023

Badatos commented Mar 22, 2023 •

edited

Loading

Badatos commented Mar 22, 2023

ptitloup commented Mar 22, 2023

ptitloup left a comment

LoicBonavent left a comment

ptitloup left a comment

[DONE] improve clean_video_files and code cleaning #782

[DONE] improve clean_video_files and code cleaning #782

Conversation

Badatos commented Mar 21, 2023 • edited Loading

gitguardian bot commented Mar 21, 2023 • edited Loading

⚠️ GitGuardian has uncovered 11 secrets following the scan of your pull request.

ptitloup left a comment

Choose a reason for hiding this comment

ptitloup left a comment

Choose a reason for hiding this comment

ptitloup commented Mar 21, 2023

Badatos commented Mar 22, 2023 • edited Loading

Badatos commented Mar 22, 2023

ptitloup commented Mar 22, 2023

ptitloup left a comment

Choose a reason for hiding this comment

LoicBonavent left a comment

Choose a reason for hiding this comment

ptitloup left a comment

Choose a reason for hiding this comment

Badatos commented Mar 21, 2023 •

edited

Loading

gitguardian bot commented Mar 21, 2023 •

edited

Loading

Badatos commented Mar 22, 2023 •

edited

Loading