Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task server Docker changes and backend tasks changes #3688

Merged
merged 1 commit into from
Apr 16, 2024
Merged

Task server Docker changes and backend tasks changes #3688

merged 1 commit into from
Apr 16, 2024

Conversation

athithyaaselvam
Copy link
Collaborator

What changes were proposed in this pull request?

Task server docker changes. Created a process hue_celery and hue_redis which is controlled by Supervisord. Added helper functions which fetches the status of tasks from redis and passes on to the react component (task server page).

How was this patch tested?

Manual testing(File uploads on HDFS, S3, Azure, Ozone. Task server page fetches the tasks. Tested scheduling and killing tasks in task server page), bug bash.

Following hue.ini configs required to test:

[desktop]
enable_chunked_file_uploader=true
[filebrowser]
max_file_size_upload_limit=-1 # Bytes
[[task_server]]
enabled=True
beat_enabled=True
fetch_result_limit=20000
result_storage='{"BACKEND": "django.core.files.storage.FileSystemStorage", "properties": {"location": "./logs"}}'
broker_url="redis://localhost:6379/0"
celery_result_backend="redis://localhost:6379/0"
celeryd_opts='--time-limit=300'
execution_storage='{"BACKEND": "django.core.cache.backends.locmem.LocMemCache", "LOCATION": "celery-hue"}'

JohanAhlen
JohanAhlen reviewed Apr 10, 2024
View reviewed changes
Copy link
Contributor

@JohanAhlen JohanAhlen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about adding some unit tests?

@athithyaaselvam athithyaaselvam mentioned this pull request Apr 10, 2024
Closed
@athithyaaselvam athithyaaselvam force-pushed the aselvam_task_server_changes2 branch 2 times, most recently from 0f5a8e5 to 78ad261 Compare April 10, 2024 17:09
Harshg999
Harshg999 reviewed Apr 15, 2024
View reviewed changes
Copy link
Collaborator

@Harshg999 Harshg999 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I've added few review comments.
Also please add unit tests

apps/filebrowser/src/filebrowser/tasks.py Show resolved Hide resolved
apps/filebrowser/src/filebrowser/tasks.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/tasks.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/tasks.py Outdated Show resolved Hide resolved
apps/filebrowser/src/filebrowser/tasks.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/views.py Outdated Show resolved Hide resolved
desktop/core/src/desktop/urls.py Show resolved Hide resolved
desktop/core/src/desktop/views.py Outdated Show resolved Hide resolved
tools/container/base/hue/Dockerfile Outdated Show resolved Hide resolved
tools/container/compile/hue/Dockerfile Outdated Show resolved Hide resolved
@athithyaaselvam
Copy link
Collaborator Author

I'll create a new PR with Unit tests and submit it.

ranade1
ranade1 approved these changes Apr 15, 2024
View reviewed changes
Copy link
Contributor

@ranade1 ranade1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@athithyaaselvam athithyaaselvam force-pushed the aselvam_task_server_changes2 branch 2 times, most recently from a4ac89d to 3cb9e75 Compare April 16, 2024 08:40
Harshg999
Harshg999 approved these changes Apr 16, 2024
View reviewed changes
Copy link
Collaborator

@Harshg999 Harshg999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Task server Docker changes and backend tasks changes
4768a52 
Change-Id: I1104a6e71530da51f4ea80f814c3bca19b59a5d6

Improving the URL routing and Task logs logic

Change-Id: I8d5bab4183950658412336ce05e1adceb6d73b1c

removing dead celery code

Change-Id: Ib1b1cfbd7dba1981658a5e645a499286d63f52a7
@athithyaaselvam athithyaaselvam enabled auto-merge (squash) April 16, 2024 17:53
@athithyaaselvam athithyaaselvam merged commit 03fcee0 into master Apr 16, 2024
3 of 4 checks passed
@athithyaaselvam athithyaaselvam deleted the aselvam_task_server_changes2 branch April 16, 2024 18:06
athithyaaselvam added a commit that referenced this pull request Jun 10, 2024
@athithyaaselvam
CDPD-64067: Task server Docker changes and backend tasks changes (#3688)
f547d25 
Change-Id: I1104a6e71530da51f4ea80f814c3bca19b59a5d6

Improving the URL routing and Task logs logic

Change-Id: I8d5bab4183950658412336ce05e1adceb6d73b1c

removing dead celery code

Change-Id: Ib1b1cfbd7dba1981658a5e645a499286d63f52a7

Co-authored-by: Athithyaa Selvam <aselvam@cloudera.com>
(cherry picked from commit 03fcee0)
Change-Id: If32cfd52c2ed6a393f56fab6837087940a08c20e
(cherry picked from commit c23977610561b3d928dc8f3e9e2f2022a8e047b6)
(cherry picked from commit 929c0cd579b8ded5e34aa48e9a29e977f09dc390)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JohanAhlen JohanAhlen JohanAhlen left review comments

@wing2fly wing2fly wing2fly approved these changes

@ranade1 ranade1 ranade1 approved these changes

@Harshg999 Harshg999 Harshg999 approved these changes

@amitsrivastava amitsrivastava Awaiting requested review from amitsrivastava

@agl29 agl29 Awaiting requested review from agl29

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@athithyaaselvam @wing2fly @JohanAhlen @ranade1 @Harshg999