Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Channel sync #5135

Merged
merged 71 commits into from
Aug 10, 2022
Merged

Channel sync #5135

merged 71 commits into from
Aug 10, 2022

Conversation

fflorent
Copy link
Contributor

@fflorent fflorent commented Jul 20, 2022

Description

Implement channel synchronization, but with some lacks detailed below.

Related issues

Fixed #754
Also it paves the way for the importation of channel videos from other platforms in the UI (like peertube-import-videos).

Has this been tested?

  • 👍 Manual tests done
  • 👍 Unit tests done

Screenshots

"My synchronizations" button

Synchronization table

Synchronization creation form

EDIT1 : Screencast of admin configuration

recording-optimized-2.mp4

Copy link
Owner

@Chocobozzz Chocobozzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this wonderful work!

Just made a first "quick" review of the code. Will do other ones in the week!

@fflorent fflorent marked this pull request as draft July 26, 2022 09:21
@fflorent fflorent force-pushed the channel-sync branch 2 times, most recently from f7ff148 to 4c5b1d8 Compare July 26, 2022 14:20
server/initializers/constants.ts Outdated Show resolved Hide resolved
server/lib/job-queue/handlers/video-channels-sync.ts Outdated Show resolved Hide resolved
config/default.yaml Outdated Show resolved Hide resolved
server/initializers/constants.ts Outdated Show resolved Hide resolved
server/lib/job-queue/handlers/video-channels-sync.ts Outdated Show resolved Hide resolved
server/lib/job-queue/handlers/video-channels-sync.ts Outdated Show resolved Hide resolved
server/lib/job-queue/handlers/video-channels-sync.ts Outdated Show resolved Hide resolved
server/controllers/api/server/debug.ts Outdated Show resolved Hide resolved
server/helpers/youtube-dl/youtube-dl-import-utils.ts Outdated Show resolved Hide resolved
server/lib/job-queue/handlers/video-channels-sync.ts Outdated Show resolved Hide resolved
support/doc/api/openapi.yaml Outdated Show resolved Hide resolved
support/doc/api/openapi.yaml Outdated Show resolved Hide resolved
support/doc/api/openapi.yaml Outdated Show resolved Hide resolved
support/doc/api/openapi.yaml Outdated Show resolved Hide resolved
@fflorent fflorent force-pushed the channel-sync branch 2 times, most recently from 1da6cb7 to f384c5a Compare August 1, 2022 20:44
@fflorent fflorent force-pushed the channel-sync branch 4 times, most recently from 7c4a47c to cbdbb67 Compare August 4, 2022 09:46
@fflorent fflorent marked this pull request as ready for review August 4, 2022 11:54
setDefaultSort,
setDefaultPagination,
asyncMiddleware(listAccountChannelsSync)
)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In another task: UI for managing other users synchronizations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue reported here #5181

server/models/video/video-channel-sync.ts Show resolved Hide resolved
Copy link
Contributor Author

@fflorent fflorent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took into account your remarks

})
}
res.locals.videoChannelSync = sync
res.locals.videoChannel = sync.VideoChannel
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need this attribute for ensureCanManageChannel which picks specifically res.locals.videoChannel.

server/models/video/video-channel-sync.ts Outdated Show resolved Hide resolved
server/models/video/video-channel-sync.ts Show resolved Hide resolved
server/initializers/checker-after-init.ts Outdated Show resolved Hide resolved
config/default.yaml Outdated Show resolved Hide resolved
server/lib/video-import-channel.ts Outdated Show resolved Hide resolved
server/models/video/video-channel-sync.ts Show resolved Hide resolved
server/initializers/constants.ts Outdated Show resolved Hide resolved
@fflorent
Copy link
Contributor Author

fflorent commented Aug 6, 2022

I improved a little bit the UI of the list of the synchronizations:
image

@fflorent fflorent force-pushed the channel-sync branch 2 times, most recently from 6a4820a to 022a2fc Compare August 8, 2022 10:10
import { synchronizeChannel } from '@server/lib/video-import-channel'

export async function processVideoChannelImport (job: Job) {
const payload = job.data as VideoChannelImportPayload
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there could not be some issues with video orders when:

  • the user creates a synchronization, requesting a channel import
  • the channel import runs while a sync is run

I wonder if we should no passe an optional syncId in the payload to flag the synchronization as being processed, and skip any of sync in PROCESSING.

What do you think @Chocobozzz?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll check how we can handle this case and update the code

@Chocobozzz
Copy link
Owner

Thanks, I'll rebase from develop and merge this PR

@Chocobozzz Chocobozzz merged commit 2a49118 into Chocobozzz:develop Aug 10, 2022
@Chocobozzz
Copy link
Owner

🎉 🎉 🎉 🙏 :)

@emansom
Copy link
Contributor

emansom commented Aug 10, 2022

I have imported a YouTube channel to my PeerTube instances a few weeks ago, when this feature wasn't available. Is there any way I can flag the existing videos as being "imported" already?

e.g. by executing queries directly on the database?

@fflorent
Copy link
Contributor Author

I have migrated a YouTube channel to my PeerTube instances a few weeks ago, when this feature wasn't available. Is there any way I can flag the existing videos as being "imported" already?

@emansom If these videos were imported through the Import with URL feature, you should have nothing to do, these videos should already be recognized as imported.

@emansom
Copy link
Contributor

emansom commented Aug 10, 2022

@emansom If these videos were imported through the Import with URL feature, you should have nothing to do, these videos should already be recognized as imported.

They were imported using the peertube-import-videos.js CLI and don't show up in the imported tab.
https://docs.joinpeertube.org/maintain-tools?id=peertube-import-videosjs

@fflorent
Copy link
Contributor Author

@emansom If these videos were imported through the Import with URL feature, you should have nothing to do, these videos should already be recognized as imported.

They were imported using the peertube-import-videos.js CLI and don't show up in the imported tab. https://docs.joinpeertube.org/maintain-tools?id=peertube-import-videosjs

I see.

That being said, if you create a new sync without importing the existing videos, only videos newer than the creation date are eligible for importation.

@emansom
Copy link
Contributor

emansom commented Aug 10, 2022

I see.

That being said, if you create a new sync without importing the existing videos, only videos newer than the creation date are eligible for importation.

Which database table and which column determines the "is imported" state? Or is it another table with video ID references that I need to add entries into?

@fflorent
Copy link
Contributor Author

I meant that you should have nothing to do, except creating a synchronization with the option "only watch for new publications". Only the videos uploaded on Youtube after the synchronizations will be imported.

If you are still willing to play with the database, the table is named videoImport and place the state value to 2. Ensure to use the sequence (videoImport_id_seq) for the id column. Note that's ok for this table if you know what you are doing, but avoid manipulating PeerTube DB tables in general (otherwise that can lead to confusion on ActivityPub).

@tio-trom
Copy link

I just want to say a huge THANK YOU! Amazing.

@Seyferto
Copy link

I meant that you should have nothing to do, except creating a synchronization with the option "only watch for new publications". Only the videos uploaded on Youtube after the synchronizations will be imported.

There's no option to import all prior ones?

@emansom
Copy link
Contributor

emansom commented Sep 11, 2022

There's no option to import all prior ones?

There is, manually. The way I did it was:

  1. Change the state as described above by @fflorent

For previous videos that weren't imported within the UI itself, but rather using the CLI tool, the following steps are required:

  1. Setting up pgAdmin and connecting to the PeerTube database
  2. Export the videos table as CSV, opening it in LibreOffice Calc and transforming it into a structure compatible with the schema of the videoImports table
  3. Importing the transformed CSV into the videoImports table using pgAdmin

Ofcourse this could be dealt with better in the form of a SQL migration script, I lack the knowledge on Postgres specific SQL to implement this.

@Seyferto
Copy link

This could also be dealt better with just an interface click that automates it... but perhaps I should get used to how mirroring and syncing will always be complex tasks, if not impossible elsewhere...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatically import videos from other platforms
5 participants