Add NoArtifactContentUploadSerializer

closes #6281

CHANGES/plugin_api/6281.feature

@@ -0,0 +1,2 @@
+Added new NoArtifactContentUploadSerializer and NoArtifactContentUploadViewSet to enable plugin
+writers to upload content without storing an Artifact

docs/components.rst

@@ -38,7 +38,7 @@ Content Serving Application
 ---------------------------
 
 An aiohttp.server based application that serves content to clients. The content could be
-:term:`Artifacts<artifact>` already downloaded and saved in Pulp, or
+:term:`Artifacts<Artifact>` already downloaded and saved in Pulp, or
 :term:`on-demand content units<on-demand content>`. When serving
 :term:`on-demand content units<on-demand content>` the downloading also happens from within this
 component as well.

docs/glossary.rst

@@ -4,7 +4,6 @@ Glossary
 .. glossary::
 
     :class:`~pulpcore.app.models.Artifact`
-    artifact
         A file. They usually belong to a :term:`content unit<Content>` but may be used
         elsewhere (e.g. for PublishedArtifacts).
 
@@ -17,7 +16,7 @@ Glossary
     content unit
         Content are the smallest units of data that can be added and removed from
         :term:`repositories<Repository>`. When singular, "content unit" should be used. Each
-        content unit can have multiple :term:`artifacts<artifact>`. Each content unit has a
+        content unit can have multiple :term:`artifacts<Artifact>`. Each content unit has a
         :term:`type` (like .rpm or .deb) which that is defined by a :term:`plugin`.
 
     content app
@@ -36,9 +35,9 @@ Glossary
 
     on-demand content
         :term:`Content<Content>` that was synchronized into Pulp but not yet saved to the
-        filesystem. The Content's :term:`Artifacts<artifact>` are fetched at the time they are
+        filesystem. The Content's :term:`Artifacts<Artifact>` are fetched at the time they are
         requested.  On-demand content is associated with a :term:`Remote` that knows how to download
-        those :term:`Artifacts<artifact>`.
+        those :term:`Artifacts<Artifact>`.
 
     plugin
         A `Django <https://docs.djangoproject.com>`_ app that exends :term:`pulpcore` to add more

docs/plugins/plugin-writer/concepts/subclassing/viewsets.rst

@@ -108,6 +108,12 @@ instead of ``validate``, this serializer can do detailed analysis of the given o
 in order to fill database fields of the content type like "name", "version", etc. This part of
 validation is only called in the task context.
 
+If the uploaded content does not need to be stored, plugin writers may derive from the class
+``NoArtifactContentUploadViewSet``. Again, the same analogy applies to this workflow. To use this
+ViewSet, the associated serializer should also subclass ``NoArtifactContentUploadSerializer``. Note
+that by using the aforesaid Serializer and ViewSet, Pulp still loads content of an uploaded file,
+if any, to a temporary Artifact. But the Artifact is going to be removed by the end of a created task.
+
 If any additional context needs to be passed from the ViewSet to the creation task, the
 ``get_deferred_context`` method of the ViewSet might be overwritten. It's return value will then be
 available as ``self.context`` in the Serializer.

docs/workflows/on-demand-downloading.rst

@@ -10,27 +10,27 @@ of the :term:`Remote` performing the sync. Policy is an optional parameter and d
 `immediate`.
 
 immediate
-  When performing the sync, download all :term:`Artifacts<artifact>` now. Also download all metadata
+  When performing the sync, download all :term:`Artifacts<Artifact>` now. Also download all metadata
   now to create the content units in Pulp, associated with the
   :term:`repository version<RepositoryVersion>` created by the sync. `immediate` is the default, and
   any plugin providing a sync is expected to implement the `immediate` mode.
 
 on_demand
-  When performing the sync, do not download any :term:`Artifacts<artifact>` now. Download all
+  When performing the sync, do not download any :term:`Artifacts<Artifact>` now. Download all
   metadata now to create the content units in Pulp, associated with the
   :term:`repository version<RepositoryVersion>` created by the sync. Clients requesting content
-  trigger the downloading of :term:`Artifacts<artifact>`, which are saved into Pulp to be served to
+  trigger the downloading of :term:`Artifacts<Artifact>`, which are saved into Pulp to be served to
   future clients.
 
   This mode is ideal for saving disk space because Pulp never downloads and stores
-  :term:`Artifacts<artifact>` that clients don't need. Units created from this mode are
+  :term:`Artifacts<Artifact>` that clients don't need. Units created from this mode are
   :term:`on-demand content units<on-demand content>`.
 
 streamed
-  When performing the sync, do not download any :term:`Artifacts<artifact>` now. Download all
+  When performing the sync, do not download any :term:`Artifacts<Artifact>` now. Download all
   metadata now to create the content units in Pulp, associated with the
   :term:`repository version<RepositoryVersion>` created by the sync. Clients requesting content
-  trigger the downloading of :term:`Artifacts<artifact>`, which are *not* saved into Pulp. This
+  trigger the downloading of :term:`Artifacts<Artifact>`, which are *not* saved into Pulp. This
   content will be re-downloaded with each client request.
 
   This mode is ideal for content that you especially don't want Pulp to store over time. For

pulpcore/app/files.py

@@ -33,6 +33,12 @@ def from_file(cls, file):
         instance = cls(name, '', file.size, '', '')
         instance.file = file
         data = file.read()
+
+        # calling the method read() moves the file's pointer to the end of the file object,
+        # thus, it is necessary to reset the file's pointer position back to 0 in case of
+        # calling the method read() again from another place
+        file.seek(0)
+
         for hasher in hashlib.algorithms_guaranteed:
             instance.hashers[hasher].update(data)
         return instance

pulpcore/app/tasks/base.py

@@ -1,5 +1,28 @@
 from pulpcore.app.apps import get_plugin_config
-from pulpcore.app.models import CreatedResource
+from pulpcore.app.models import Artifact, CreatedResource
+from pulpcore.app.files import PulpTemporaryUploadedFile
+
+
+def general_create_from_temp_file(app_label, serializer_name, *args, **kwargs):
+    """
+    Create a model instance from contents stored in a temporary Artifact.
+
+    A caller should always pass the dictionary "data", as a keyword argument, containing the
+    href to the temporary Artifact. Otherwise, the function does nothing.
+
+    This function calls the function general_create() to create a model instance.
+    Data passed to that function already contains a serialized artifact converted
+    to PulpTemporaryUploadFile that will be deleted afterwards.
+    """
+    data = kwargs.pop("data", None)
+    if data and "artifact" in data:
+        named_model_view_set = get_plugin_config(app_label).viewsets_module.NamedModelViewSet
+        artifact = named_model_view_set.get_resource(data.pop("artifact"), Artifact)
+
+        data["file"] = PulpTemporaryUploadedFile.from_file(artifact.file)
+
+        general_create(app_label, serializer_name, data=data, *args, **kwargs)
+        artifact.delete()
 
 
 def general_create(app_label, serializer_name, *args, **kwargs):

pulpcore/plugin/serializers/__init__.py

@@ -28,4 +28,7 @@
     validate_unknown_fields,
 )
 
-from .content import SingleArtifactContentUploadSerializer  # noqa
+from .content import (  # noqa
+    NoArtifactContentUploadSerializer,
+    SingleArtifactContentUploadSerializer,
+)

pulpcore/plugin/serializers/content.py

@@ -4,29 +4,26 @@
 
 from rest_framework.serializers import (
     FileField,
+    Serializer,
     ValidationError,
 )
 from pulpcore.plugin.models import Artifact, Repository
-from pulpcore.plugin.serializers import DetailRelatedField, SingleArtifactContentSerializer
+from pulpcore.plugin.serializers import (
+    DetailRelatedField,
+    NoArtifactContentSerializer,
+    SingleArtifactContentSerializer,
+)
 
 
 log = getLogger(__name__)
 
 
-class SingleArtifactContentUploadSerializer(SingleArtifactContentSerializer):
-    """
-    A serializer for content_types with a single Artifact.
-
-    The Artifact can either be specified via it's url, or a new file can be uploaded.
-    Additionally a repository can be specified, to which the content unit will be added.
-
-    When using this serializer, the creation of the real content must be wrapped in a task that
-    locks the Artifact and when specified, the repository.
-    """
+class UploadSerializerFieldsMixin(Serializer):
+    """A mixin class that contains fields and methods common to content upload serializers."""
 
     file = FileField(
         help_text=_(
-            "An uploaded file that should be turned into the artifact of the content unit."
+            "An uploaded file that may be turned into the artifact of the content unit."
         ),
         required=False,
         write_only=True,
@@ -40,6 +37,54 @@ class SingleArtifactContentUploadSerializer(SingleArtifactContentSerializer):
         queryset=Repository.objects.all(),
     )
 
+    def create(self, validated_data):
+        """
+        Save a GenericContent unit.
+
+        This must be used inside a task that locks on the Artifact and if given, the repository.
+        """
+
+        repository = validated_data.pop("repository", None)
+        content = super().create(validated_data)
+
+        if repository:
+            repository.cast()
+            content_to_add = self.Meta.model.objects.filter(pk=content.pk)
+
+            # create new repo version with uploaded package
+            with repository.new_version() as new_version:
+                new_version.add_content(content_to_add)
+        return content
+
+    class Meta:
+        fields = ('file', 'repository')
+
+
+class NoArtifactContentUploadSerializer(UploadSerializerFieldsMixin, NoArtifactContentSerializer):
+    """A serializer for content types with no Artifact."""
+
+    def create(self, validated_data):
+        """Create a new content and remove the already parsed file from validated_data."""
+        validated_data.pop("file", None)
+        return super().create(validated_data)
+
+    class Meta:
+        fields = NoArtifactContentSerializer.Meta.fields \
+                 + UploadSerializerFieldsMixin.Meta.fields
+
+
+class SingleArtifactContentUploadSerializer(UploadSerializerFieldsMixin,
+                                            SingleArtifactContentSerializer):
+    """
+    A serializer for content_types with a single Artifact.
+
+    The Artifact can either be specified via it's url, or a new file can be uploaded.
+    Additionally a repository can be specified, to which the content unit will be added.
+
+    When using this serializer, the creation of the real content must be wrapped in a task that
+    locks the Artifact and when specified, the repository.
+    """
+
     def __init__(self, *args, **kwargs):
         """Initializer for SingleArtifactContentUploadSerializer."""
         super().__init__(*args, **kwargs)
@@ -66,7 +111,8 @@ def validate(self, data):
         return data
 
     def deferred_validate(self, data):
-        """Validate the content unit by deeply analyzing the specified Artifact.
+        """
+        Validate the content unit by deeply analyzing the specified Artifact.
 
         This is only called when validating without a request context to prevent stalling
         an ongoing http request.
@@ -75,22 +121,6 @@ def deferred_validate(self, data):
         """
         return data
 
-    def create(self, validated_data):
-        """Save the GenericContent unit.
-        This must be used inside a task that locks on the Artifact and if given, the repository.
-        """
-
-        repository = validated_data.pop("repository", None)
-        content = super().create(validated_data)
-
-        if repository:
-            repository.cast()
-            content_to_add = self.Meta.model.objects.filter(pk=content.pk)
-
-            # create new repo version with uploaded package
-            with repository.new_version() as new_version:
-                new_version.add_content(content_to_add)
-        return content
-
     class Meta(SingleArtifactContentSerializer.Meta):
-        fields = SingleArtifactContentSerializer.Meta.fields + ("file", "repository")
+        fields = SingleArtifactContentSerializer.Meta.fields \
+                 + UploadSerializerFieldsMixin.Meta.fields

pulpcore/plugin/viewsets/__init__.py

@@ -28,4 +28,7 @@
     RepositoryVersionFilter,
 )
 
-from .content import SingleArtifactContentUploadViewSet  # noqa
+from .content import (  # noqa
+    NoArtifactContentUploadViewSet,
+    SingleArtifactContentUploadViewSet,
+)