Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Suppression des ressources communautaires orphelines depuis l'application #1685

Merged
merged 36 commits into from Jul 6, 2021
Merged

Suppression des ressources communautaires orphelines depuis l'application #1685

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6b14acf
create CommunityResourcesCleaner module
fchabouis Jun 24, 2021
4659dc3
add some tests to list orphan resources
fchabouis Jun 24, 2021
4eb73b1
move transport publisher label to config
fchabouis Jun 24, 2021
b64fc0b
http 204 (no content) is also ok
fchabouis Jun 28, 2021
3929330
add the delete method to datagouvfr client api
fchabouis Jun 28, 2021
6bfdfc9
tricky code simplification
fchabouis Jun 28, 2021
32e4aa6
theoritically "delete" could also yield a 202 resp
fchabouis Jun 28, 2021
56253b2
format
fchabouis Jun 28, 2021
fdc5964
function to create headers
fchabouis Jun 28, 2021
1f52629
add the "delete" part in the cleaning process
fchabouis Jun 28, 2021
6259538
create A Mock for Datagouvfr.Client.CommunityResources
fchabouis Jun 28, 2021
843513a
say in config which implementation of CommunityResources to use
fchabouis Jun 28, 2021
6b32f49
typo
fchabouis Jun 28, 2021
04161de
test the delete part of the orphan cleaning
fchabouis Jun 28, 2021
e70161e
add a task in the scheduler for orphans
fchabouis Jun 28, 2021
352ce12
Merge branch 'master' into resources_cleanup
fchabouis Jun 28, 2021
baf0fb2
update tests of orphan detection
fchabouis Jun 28, 2021
0c3bd25
format file
fchabouis Jun 28, 2021
9d5f64e
specify async true for the cleaner tests
fchabouis Jun 28, 2021
56160ef
Merge remote-tracking branch 'origin/resources_cleanup' into resource…
fchabouis Jun 28, 2021
718656e
try with async false
fchabouis Jun 28, 2021
ac35037
its not a Mock, its a Stub
fchabouis Jun 29, 2021
4d9102f
add Mox dependency in datagouvfr
fchabouis Jun 29, 2021
a7a5774
define a Mock for CommunityResource module
fchabouis Jun 29, 2021
fe36a17
use the CommunityResources stub here
fchabouis Jun 29, 2021
4b019e6
Use CommunityResources.Mock for backoffice testing
fchabouis Jun 29, 2021
db64dc5
use the stub for the test
fchabouis Jun 29, 2021
3d1748d
add an insert on the dataset_id
fchabouis Jun 29, 2021
7339567
use CommunityResources stub by default for some tests
fchabouis Jun 29, 2021
c9ea239
set uniform case for Datagouvfr module
fchabouis Jun 30, 2021
f0dc7d5
Merge branch 'master' into resources_cleanup
fchabouis Jul 5, 2021
6c88efe
log succes and errors counts
fchabouis Jul 5, 2021
292535a
datagouv_id should be passed for deletion
fchabouis Jul 5, 2021
2171a3a
update the tests
fchabouis Jul 5, 2021
588f5d9
correct bug to get error_n and ok_n values
fchabouis Jul 5, 2021
7422fe6
Merge branch 'master' into resources_cleanup
fchabouis Jul 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions apps/datagouvfr/lib/datagouvfr/client/community_resources.ex
View file
Open in desktop
Expand Up @@ -16,7 +16,7 @@ end

defmodule Datagouvfr.Client.CommunityResources.API do
@moduledoc """
Actual implementation to intercat with community resources through data.gouv.fr API
Actual implementation to interact with community resources through data.gouv.fr API
"""
require Logger

Expand Down Expand Up @@ -54,9 +54,9 @@ defmodule Datagouvfr.Client.CommunityResources.API do
end
end

defmodule Datagouvfr.Client.CommunityResources.Mock do
defmodule Datagouvfr.Client.StubCommunityResources do
@moduledoc """
A mock used for testing
A stub used for testing, when we don't really care about community resources
"""
@behaviour Datagouvfr.Client.CommunityResources

Expand Down
3 changes: 2 additions & 1 deletion apps/datagouvfr/mix.exs
View file
Open in desktop
Expand Up @@ -36,7 +36,8 @@ defmodule Datagouvfr.MixProject do
{:plug, ">= 0.0.0"},
{:shared, in_umbrella: true},
{:vex, "~> 0.8"},
{:exvcr, "~> 0.10", only: :test}
{:exvcr, "~> 0.10", only: :test},
{:mox, "~> 1.0.0", only: :test}
]
end
end
1 change: 1 addition & 0 deletions apps/datagouvfr/test/support/mocks.ex
View file
Open in desktop
@@ -0,0 +1 @@
Mox.defmock(Datagouvfr.Client.CommunityResources.Mock, for: Datagouvfr.Client.CommunityResources)
1 change: 1 addition & 0 deletions apps/transport/test/transport/community_resource_cleaner_test.exs
View file
Open in desktop
Expand Up @@ -7,6 +7,7 @@ defmodule Transport.CommunityResourcesCleanerTest do
@transport_publisher_label Application.get_env(:transport, :datagouvfr_transport_publisher_label)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Attention ces références en tant que module ça va marcher dans les "exs" mais la prochaine version de Credo va bloquer ces appels. Il faut remplacer par quelque chose qui n'est pas évalué à la compilation.


setup do
Mox.stub_with(Datagouvfr.Client.CommunityResources.Mock, Datagouvfr.Client.StubCommunityResources)
:ok = Ecto.Adapters.SQL.Sandbox.checkout(DB.Repo)
end

Expand Down
40 changes: 40 additions & 0 deletions apps/transport/test/transport_web/controllers/backoffice_controller_test.exs
View file
Open in desktop
Expand Up @@ -5,6 +5,9 @@ defmodule TransportWeb.BackofficeControllerTest do
alias DB.{Repo, Resource}
import Ecto.Repo

import Mox
setup :verify_on_exit!

@dataset_url "https://demo.data.gouv.fr/fr/datasets/horaires-theoriques-du-reseau-de-transport-tag-1/"
@dataset %{
"url" => @dataset_url,
Expand Down Expand Up @@ -110,6 +113,24 @@ defmodule TransportWeb.BackofficeControllerTest do
|> Map.put("region_id", Repo.get_by(Region, nom: "Auvergne-Rhône-Alpes").id)
|> Map.put("insee", nil)

Datagouvfr.Client.CommunityResources.Mock
|> expect(:get, fn _id ->
# we return the same urls that the one we find in dataset-region.json cassette
# because for the moment the Hasher is not Mocked
# we it is the case, we will be able to put random urls here
{:ok,
[
%{
"url" => "https://app-be8e53a7-9b77-4f95-bea0-681b97077017.cleverapps.io/metromobilite/gtfs-rt.json",
"id" => "r1"
},
%{
"url" => "https://app-be8e53a7-9b77-4f95-bea0-681b97077017.cleverapps.io/metromobilite/gtfs-rt",
"id" => "r2"
}
]}
end)

conn =
use_cassette "dataset/dataset-region.json" do
post(conn, backoffice_dataset_path(conn, :post), dataset)
Expand All @@ -132,6 +153,25 @@ defmodule TransportWeb.BackofficeControllerTest do

dataset = %{@dataset | "region_id" => nil}

Datagouvfr.Client.CommunityResources.Mock
|> expect(:get, fn _id ->
# we return the same urls that the one we find in dataset-aom.json cassette
fchabouis marked this conversation as resolved.
Show resolved Hide resolved
# because for the moment the Hasher is not Mocked
# we it is the case, we will be able to put random urls here

{:ok,
[
%{
"url" => "https://app-be8e53a7-9b77-4f95-bea0-681b97077017.cleverapps.io/metromobilite/gtfs-rt.json",
"id" => "r1"
},
%{
"url" => "https://app-be8e53a7-9b77-4f95-bea0-681b97077017.cleverapps.io/metromobilite/gtfs-rt",
"id" => "r2"
}
]}
end)

conn =
use_cassette "dataset/dataset-aom.json" do
post(conn, backoffice_dataset_path(conn, :post), dataset)
Expand Down