Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fully working sync #32

Merged
merged 76 commits into from
Feb 21, 2024

Conversation

EmberLightVFX
Copy link
Contributor

@EmberLightVFX EmberLightVFX commented Feb 16, 2024

This PR is a continuation on @scottmcdonnell PR #31 (which in case is a continuation of #28)
I made a fork of his fork and pushed a PR to his but I haven't heard anything from him in a week so I decided to push this PR directly to the Ayon repository instead.

This PR completes all expected sync-posibilities from Kitsu to Ayon.
All of these entities support full_sync, create, edit and delete events

  • Assets
  • Episodes
  • Sequences
  • Shots
  • Edits
  • Users
  • Assignees to tasks

Project attributes can be edited and projects can also deleted if enabled in settings (disabled by default).

Usernames are generated using {first_name}.{laste_name} lowercased, so Jacob Danell would end up being jacob.danell
The email is synced from kitsu.
The password gets a default password that's being able to be set in the settings as only admins currently can set passwords for accounts without passwords?

For entities with comments now sync their comments.
For shots it also calculates the in/out frame from the metadata.

The code to sync Concepts exist but the current zou version (0.18.7) have a bug where Concept events also executes Assets events that messes things up for us, so that code is commented out and when CGWire have fixed the problem we just need to un-comment the code.

I have also added settings for all new syncing features:

  • Delete projects (on/off)
  • Sync users (on/off)
    • Default password
    • Access group
    • Roles (to convert Kitsu roles to Ayon roles)
  • Default sync info
    • Tasks
      • Name
      • Short name
      • Icon
    • Statuses
      • Short name
      • State
      • Icon

The files I have done any changes with have bin formatted with black, added type annotations, fixed bugs and cleaned up unused codes and imports.

It also includes an updated create_package.py that allows you to automatically upload the plugin to Ayon and reload the server. Super great for fast development :)

The only thing that I couldn't get working is if the user updates an asset-type, task-type or task-status. In kitsu these are project independent. In Ayon these doesn't have any data dict in their instance so it's impossible to match anything that might have changed.

scott and others added 30 commits December 19, 2023 20:49
…ks to be skipped, for the moment, to completed the syncing.
…nd the sync will continue otherwise the sync stops at the first exception.
# Conflicts:
#	server/kitsu/push.py
#	version.py
…task.name

test for create task.status and test for updating tasks
all tests passing
tests for create, update and delete.
tested with live kitsu and processor working
@EmberLightVFX EmberLightVFX marked this pull request as ready for review February 16, 2024 17:10
@martastain martastain self-requested a review February 19, 2024 08:22
@martastain
Copy link
Member

This is a great job! I am still testing, but it looks super promising. I really appreciate the code readability. Just two notes so far:

Do you think you could move tests to a separate directory as mentioned here? #21 (comment) I think having a pyproject.toml handling just tests in the addon root is a little confusing.

When i tried it with older kitsu (0.17.7) processor throws the following exceptions, which is probably expected, but maybe silent catching would make sense for studios with older kitsu?

    Traceback (most recent call last):
      File "/service/processor/processor.py", line 325, in start_processing
        full_sync(self, kitsu_project_id, ayon_project_name)
      File "/service/processor/fullsync.py", line 59, in full_sync
        concepts = gazu.concept.all_concepts_for_project(kitsu_project_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/gazu/cache.py", line 204, in wrapper
        return function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/gazu/concept.py", line 33, in all_concepts_for_project
        concepts = raw.fetch_all(
                   ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/gazu/client.py", line 339, in fetch_all
        return get(url_path_join("data", path), params=params, client=client)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/gazu/client.py", line 206, in get
        check_status(response, path)
      File "/usr/local/lib/python3.11/site-packages/gazu/client.py", line 298, in check_status
        raise RouteNotFoundException(path)
    gazu.exception.RouteNotFoundException: data/projects/fd7bdc19-fdd9-4755-a357-8f8be7c56f4d/concepts

@EmberLightVFX
Copy link
Contributor Author

Do you think you could move tests to a separate directory as mentioned here? #21 (comment) I think having a pyproject.toml handling just tests in the addon root is a little confusing.

Done! I also updated the test-readme, fixed some test-bugs I found and cleaned up the test-code :)

@EmberLightVFX
Copy link
Contributor Author

When i tried it with older kitsu (0.17.7) processor throws the following exceptions, which is probably expected, but maybe silent catching would make sense for studios with older kitsu?

Nice find! Concepts were introduced in Kitsu/Zou 0.18.0
Didn't think about this one. I have wrapped concepts around an try/except so you shouldn't get any errors now :)

@EmberLightVFX
Copy link
Contributor Author

cgwire just fixed the concept event listening in Zou 0.19.0 so I added a check for if the user is running Zou 0.19 or above, in that case start listening for Concepts events :)

Copy link
Member

@martastain martastain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great. Any chance for @64qam or @m-u-r-p-h-y to test it as well?

I will have some notes for the user mapping, but since that is a completely new feature, it at this point does not break anything and we could look into it together in one of the upcoming releases.

Copy link
Member

@m-u-r-p-h-y m-u-r-p-h-y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested

  • creating shots
  • removing tasks
  • adding tasks

everything is instantly synced from Kitsu to Ayon

image

@martastain martastain merged commit 516807b into ynput:develop Feb 21, 2024
@EmberLightVFX EmberLightVFX deleted the enhancement/fully_working_sync branch February 21, 2024 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants